Oct 30 20:37:19.103124 ip-10-0-130-52 systemd[1]: Starting Kubernetes Kubelet... Oct 30 20:37:19.482202 ip-10-0-130-52 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:37:19.482202 ip-10-0-130-52 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 20:37:19.482202 ip-10-0-130-52 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:37:19.482202 ip-10-0-130-52 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:37:19.485355 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485275 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:37:19.485355 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485355 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:37:19.485422 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485359 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:37:19.485422 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485363 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:37:19.485422 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485365 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:37:19.485422 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485369 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:37:19.485422 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485371 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:37:19.485422 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485374 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:37:19.485422 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485378 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:37:19.485422 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485381 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:37:19.485422 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485386 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:37:19.485422 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485389 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:37:19.485422 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485392 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:37:19.485422 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485395 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:37:19.485422 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485398 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:37:19.485422 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485401 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:37:19.485422 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485403 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:37:19.485422 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485406 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:37:19.485422 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485409 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:37:19.485422 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485411 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:37:19.485422 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485414 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:37:19.485856 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485423 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:37:19.485856 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485426 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:37:19.485856 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485429 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:37:19.485856 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485432 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:37:19.485856 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485435 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:37:19.485856 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485438 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:37:19.485856 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485441 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:37:19.485856 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485443 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:37:19.485856 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485446 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:37:19.485856 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485449 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:37:19.485856 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485451 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:37:19.485856 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485454 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:37:19.485856 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485457 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:37:19.485856 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485459 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:37:19.485856 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485462 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:37:19.485856 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485464 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:37:19.485856 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485467 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:37:19.485856 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485470 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:37:19.485856 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485472 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:37:19.485856 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485475 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:37:19.486326 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485477 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:37:19.486326 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485480 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:37:19.486326 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485482 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:37:19.486326 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485485 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:37:19.486326 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485487 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:37:19.486326 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485490 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:37:19.486326 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485493 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:37:19.486326 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485495 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:37:19.486326 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485498 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:37:19.486326 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485500 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:37:19.486326 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485502 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:37:19.486326 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485506 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:37:19.486326 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485508 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:37:19.486326 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485511 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:37:19.486326 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485514 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:37:19.486326 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485517 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:37:19.486326 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485520 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:37:19.486326 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485522 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:37:19.486326 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485525 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:37:19.486326 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485527 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:37:19.486800 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485530 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:37:19.486800 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485533 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:37:19.486800 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485535 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:37:19.486800 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485538 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:37:19.486800 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485541 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:37:19.486800 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485544 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:37:19.486800 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485546 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:37:19.486800 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485549 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:37:19.486800 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485552 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:37:19.486800 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485554 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:37:19.486800 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485557 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:37:19.486800 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485560 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:37:19.486800 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485563 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:37:19.486800 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485565 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:37:19.486800 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485568 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:37:19.486800 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485571 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:37:19.486800 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485574 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:37:19.486800 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485576 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:37:19.486800 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485579 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:37:19.487253 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485581 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:37:19.487253 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485584 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:37:19.487253 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485586 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:37:19.487253 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485589 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:37:19.487253 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485591 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:37:19.487253 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485594 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:37:19.487253 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485598 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:37:19.487253 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485600 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:37:19.487253 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485603 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:37:19.487253 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485606 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:37:19.487253 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485608 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:37:19.487253 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485611 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:37:19.487253 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485613 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:37:19.487253 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485616 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:37:19.487253 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485618 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:37:19.487253 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485621 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:37:19.487253 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485623 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:37:19.487253 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485626 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:37:19.487253 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485629 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:37:19.487729 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485634 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:37:19.487729 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485638 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:37:19.487729 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485641 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:37:19.487729 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485644 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:37:19.487729 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485647 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:37:19.487729 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485649 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:37:19.487729 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.485653 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:37:19.487729 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486437 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:37:19.487729 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486444 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:37:19.487729 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486447 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:37:19.487729 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486450 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:37:19.487729 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486453 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:37:19.487729 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486455 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:37:19.487729 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486458 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:37:19.487729 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486461 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:37:19.487729 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486463 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:37:19.487729 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486466 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:37:19.487729 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486468 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:37:19.487729 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486472 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:37:19.488194 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486476 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:37:19.488194 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486479 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:37:19.488194 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486481 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:37:19.488194 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486484 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:37:19.488194 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486487 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:37:19.488194 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486490 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:37:19.488194 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486492 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:37:19.488194 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486495 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:37:19.488194 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486498 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:37:19.488194 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486500 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:37:19.488194 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486503 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:37:19.488194 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486506 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:37:19.488194 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486509 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:37:19.488194 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486511 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:37:19.488194 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486514 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:37:19.488194 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486516 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:37:19.488194 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486519 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:37:19.488194 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486521 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:37:19.488194 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486524 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:37:19.488647 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486527 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:37:19.488647 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486531 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:37:19.488647 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486534 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:37:19.488647 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486536 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:37:19.488647 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486539 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:37:19.488647 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486541 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:37:19.488647 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486544 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:37:19.488647 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486546 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:37:19.488647 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486549 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:37:19.488647 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486551 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:37:19.488647 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486556 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:37:19.488647 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486560 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:37:19.488647 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486563 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:37:19.488647 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486567 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:37:19.488647 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486570 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:37:19.488647 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486574 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:37:19.488647 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486578 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:37:19.488647 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486581 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:37:19.488647 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486584 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:37:19.489117 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486587 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:37:19.489117 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486589 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:37:19.489117 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486592 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:37:19.489117 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486595 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:37:19.489117 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486597 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:37:19.489117 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486600 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:37:19.489117 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486602 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:37:19.489117 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486605 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:37:19.489117 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486607 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:37:19.489117 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486610 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:37:19.489117 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486613 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:37:19.489117 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486615 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:37:19.489117 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486618 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:37:19.489117 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486620 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:37:19.489117 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486623 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:37:19.489117 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486626 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:37:19.489117 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486629 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:37:19.489117 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486632 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:37:19.489117 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486635 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:37:19.489117 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486638 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:37:19.489594 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486641 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:37:19.489594 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486643 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:37:19.489594 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486646 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:37:19.489594 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486648 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:37:19.489594 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486651 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:37:19.489594 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486653 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:37:19.489594 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486656 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:37:19.489594 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486659 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:37:19.489594 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486661 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:37:19.489594 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486663 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:37:19.489594 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486667 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:37:19.489594 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486670 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:37:19.489594 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486672 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:37:19.489594 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486675 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:37:19.489594 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486677 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:37:19.489594 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486679 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:37:19.489594 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486682 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:37:19.489594 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486684 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:37:19.489594 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486687 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:37:19.489594 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486689 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:37:19.490088 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486692 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:37:19.490088 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486694 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:37:19.490088 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486697 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:37:19.490088 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486699 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:37:19.490088 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486702 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:37:19.490088 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486704 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:37:19.490088 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486707 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:37:19.490088 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486709 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:37:19.490088 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486712 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:37:19.490088 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486715 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:37:19.490088 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486717 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:37:19.490088 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486720 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:37:19.490088 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486723 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:37:19.490088 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486725 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:37:19.490088 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486728 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:37:19.490088 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.486731 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:37:19.490088 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487245 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:37:19.490088 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487252 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:37:19.490088 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487256 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:37:19.490088 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487259 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:37:19.490565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487262 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:37:19.490565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487265 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:37:19.490565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487268 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:37:19.490565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487271 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:37:19.490565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487274 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:37:19.490565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487277 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:37:19.490565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487279 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:37:19.490565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487282 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:37:19.490565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487285 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:37:19.490565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487287 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:37:19.490565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487289 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:37:19.490565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487292 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:37:19.490565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487294 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:37:19.490565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487297 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:37:19.490565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487300 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:37:19.490565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487302 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:37:19.490565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487305 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:37:19.490565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487307 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:37:19.490565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487310 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:37:19.490565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487313 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:37:19.491055 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487316 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:37:19.491055 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487319 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:37:19.491055 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487321 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:37:19.491055 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487324 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:37:19.491055 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487327 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:37:19.491055 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487330 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:37:19.491055 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487333 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:37:19.491055 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487336 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:37:19.491055 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487340 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:37:19.491055 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487344 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:37:19.491055 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487347 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:37:19.491055 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487350 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:37:19.491055 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487353 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:37:19.491055 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487355 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:37:19.491055 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487359 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:37:19.491055 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487363 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:37:19.491055 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487366 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:37:19.491055 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487369 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:37:19.491055 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487372 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:37:19.491508 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487374 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:37:19.491508 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487377 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:37:19.491508 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487380 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:37:19.491508 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487382 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:37:19.491508 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487385 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:37:19.491508 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487387 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:37:19.491508 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487390 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:37:19.491508 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487392 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:37:19.491508 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487395 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:37:19.491508 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487397 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:37:19.491508 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487400 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:37:19.491508 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487402 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:37:19.491508 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487405 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:37:19.491508 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487407 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:37:19.491508 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487411 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:37:19.491508 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487414 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:37:19.491508 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487416 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:37:19.491508 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487419 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:37:19.491508 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487422 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:37:19.491977 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487424 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:37:19.491977 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487427 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:37:19.491977 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487430 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:37:19.491977 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487433 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:37:19.491977 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487435 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:37:19.491977 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487438 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:37:19.491977 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487440 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:37:19.491977 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487443 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:37:19.491977 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487446 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:37:19.491977 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487448 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:37:19.491977 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487451 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:37:19.491977 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487453 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:37:19.491977 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487456 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:37:19.491977 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487458 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:37:19.491977 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487461 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:37:19.491977 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487463 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:37:19.491977 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487466 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:37:19.491977 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487468 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:37:19.491977 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487471 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:37:19.491977 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487473 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:37:19.492477 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487476 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:37:19.492477 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487478 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:37:19.492477 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487480 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:37:19.492477 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487483 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:37:19.492477 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487485 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:37:19.492477 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487488 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:37:19.492477 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487491 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:37:19.492477 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487497 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:37:19.492477 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487500 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:37:19.492477 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487502 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:37:19.492477 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487505 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:37:19.492477 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487507 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:37:19.492477 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487512 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:37:19.492477 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487515 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:37:19.492477 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487517 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:37:19.492477 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487520 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:37:19.492477 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487522 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:37:19.492477 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487525 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:37:19.492477 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487527 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487530 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487533 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487535 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487538 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.487540 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487648 2580 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487656 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487664 2580 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487668 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487673 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487677 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487681 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487686 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487689 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487692 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487695 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487699 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487702 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487705 2580 flags.go:64] FLAG: --cgroup-root="" Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487708 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487713 2580 flags.go:64] FLAG: --client-ca-file="" Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487716 2580 flags.go:64] FLAG: --cloud-provider="external" Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487720 2580 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 20:37:19.492939 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487724 2580 flags.go:64] FLAG: --cluster-domain="" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487727 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487735 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487739 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487743 2580 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487752 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487755 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487758 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487762 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487765 2580 flags.go:64] FLAG: --contention-profiling="false" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487769 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487772 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487775 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487778 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487783 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487786 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487789 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487792 2580 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487795 2580 flags.go:64] FLAG: --enable-server="true" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487798 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487803 2580 flags.go:64] FLAG: --event-burst="100" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487806 2580 flags.go:64] FLAG: --event-qps="50" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487809 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487813 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 20:37:19.493501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487816 2580 flags.go:64] FLAG: --eviction-hard="" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487821 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487824 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487827 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487830 2580 flags.go:64] FLAG: --eviction-soft="" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487834 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487838 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487841 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487844 2580 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487847 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487851 2580 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487855 2580 flags.go:64] FLAG: --feature-gates="" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487871 2580 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487875 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487878 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487881 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487884 2580 flags.go:64] FLAG: --healthz-port="10248" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487888 2580 flags.go:64] FLAG: --help="false" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487891 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487894 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487897 2580 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487900 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487904 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 20:37:19.494105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487907 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487910 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487912 2580 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487916 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487919 2580 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487922 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487925 2580 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487927 2580 flags.go:64] FLAG: --kube-reserved="" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487931 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487933 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487937 2580 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487940 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487943 2580 flags.go:64] FLAG: --lock-file="" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487945 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487963 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487967 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487973 2580 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487977 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487981 2580 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487986 2580 flags.go:64] FLAG: --logging-format="text" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487989 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487992 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487995 2580 flags.go:64] FLAG: --manifest-url="" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.487998 2580 flags.go:64] FLAG: --manifest-url-header="" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488003 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 20:37:19.494650 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488006 2580 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488010 2580 flags.go:64] FLAG: --max-pods="110" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488013 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488016 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488019 2580 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488022 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488026 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488028 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488031 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488039 2580 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488042 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488045 2580 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488049 2580 flags.go:64] FLAG: --pod-cidr="" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488052 2580 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488055 2580 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488059 2580 flags.go:64] FLAG: --pods-per-core="0" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488062 2580 flags.go:64] FLAG: --port="10250" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488065 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488067 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-01592fd9ea0c03c56" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488071 2580 flags.go:64] FLAG: --qos-reserved="" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488074 2580 flags.go:64] FLAG: --read-only-port="10255" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488078 2580 flags.go:64] FLAG: --register-node="true" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488081 2580 flags.go:64] FLAG: --register-with-taints="" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488085 2580 flags.go:64] FLAG: --registry-burst="10" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488088 2580 flags.go:64] FLAG: --registry-qps="5" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488091 2580 flags.go:64] FLAG: --reserved-cpus="" Oct 30 20:37:19.495248 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488096 2580 flags.go:64] FLAG: --reserved-memory="" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488100 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488103 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488107 2580 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488110 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488113 2580 flags.go:64] FLAG: --runonce="false" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488116 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488119 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488122 2580 flags.go:64] FLAG: --seccomp-default="false" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488125 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488128 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488131 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488134 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488137 2580 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488140 2580 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488143 2580 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488146 2580 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488149 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488152 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488155 2580 flags.go:64] FLAG: --system-cgroups="" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488158 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488164 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488166 2580 flags.go:64] FLAG: --tls-cert-file="" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488169 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488173 2580 flags.go:64] FLAG: --tls-min-version="" Oct 30 20:37:19.495910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488176 2580 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488179 2580 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488185 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488187 2580 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488190 2580 flags.go:64] FLAG: --v="2" Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488196 2580 flags.go:64] FLAG: --version="false" Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488201 2580 flags.go:64] FLAG: --vmodule="" Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488207 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488211 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488356 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488360 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488363 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488367 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488370 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488372 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488375 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488378 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488381 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488384 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488387 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488389 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488392 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:37:19.496480 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488395 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:37:19.497018 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488398 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:37:19.497018 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488400 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:37:19.497018 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488403 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:37:19.497018 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488406 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:37:19.497018 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488409 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:37:19.497018 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488411 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:37:19.497018 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488414 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:37:19.497018 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488416 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:37:19.497018 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488419 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:37:19.497018 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488421 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:37:19.497018 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488424 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:37:19.497018 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488428 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:37:19.497018 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488431 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:37:19.497018 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488433 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:37:19.497018 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488436 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:37:19.497018 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488439 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:37:19.497018 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488442 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:37:19.497018 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488445 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:37:19.497018 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488448 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:37:19.497468 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488451 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:37:19.497468 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488454 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:37:19.497468 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488456 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:37:19.497468 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488459 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:37:19.497468 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488461 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:37:19.497468 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488464 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:37:19.497468 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488467 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:37:19.497468 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488469 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:37:19.497468 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488472 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:37:19.497468 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488474 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:37:19.497468 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488477 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:37:19.497468 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488479 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:37:19.497468 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488482 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:37:19.497468 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488484 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:37:19.497468 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488487 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:37:19.497468 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488491 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:37:19.497468 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488495 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:37:19.497468 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488498 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:37:19.497468 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488501 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:37:19.497931 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488503 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:37:19.497931 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488506 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:37:19.497931 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488508 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:37:19.497931 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488511 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:37:19.497931 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488513 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:37:19.497931 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488517 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:37:19.497931 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488520 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:37:19.497931 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488522 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:37:19.497931 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488525 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:37:19.497931 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488528 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:37:19.497931 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488531 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:37:19.497931 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488534 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:37:19.497931 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488537 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:37:19.497931 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488540 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:37:19.497931 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488543 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:37:19.497931 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488545 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:37:19.497931 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488548 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:37:19.497931 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488551 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:37:19.497931 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488554 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:37:19.497931 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488557 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:37:19.498411 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488559 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:37:19.498411 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488562 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:37:19.498411 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488564 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:37:19.498411 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488567 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:37:19.498411 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488569 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:37:19.498411 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488572 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:37:19.498411 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488574 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:37:19.498411 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488578 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:37:19.498411 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488582 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:37:19.498411 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488586 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:37:19.498411 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488589 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:37:19.498411 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488592 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:37:19.498411 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488594 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:37:19.498411 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488597 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:37:19.498411 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488600 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:37:19.498411 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488603 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:37:19.498411 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488606 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:37:19.498411 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488610 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:37:19.498845 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488613 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:37:19.498845 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488615 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:37:19.498845 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488618 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:37:19.498845 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488620 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:37:19.498845 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488625 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:37:19.498845 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488627 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:37:19.498845 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488630 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:37:19.498845 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488633 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:37:19.498845 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488636 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:37:19.498845 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488638 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:37:19.498845 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488641 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:37:19.498845 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488644 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:37:19.498845 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488646 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:37:19.498845 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488649 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:37:19.498845 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488652 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:37:19.498845 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488655 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:37:19.498845 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.488680 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:37:19.499348 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488719 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:37:19.499348 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488723 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:37:19.499348 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488727 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:37:19.499348 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488730 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:37:19.499348 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488733 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:37:19.499348 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488736 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:37:19.499348 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488738 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:37:19.499348 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488743 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:37:19.499348 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488747 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:37:19.499348 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488750 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:37:19.499348 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488753 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:37:19.499348 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488756 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:37:19.499348 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488758 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:37:19.499348 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488761 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:37:19.499348 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488764 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:37:19.499348 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488768 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:37:19.499348 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488771 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:37:19.499348 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488773 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:37:19.499785 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488776 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:37:19.499785 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488779 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:37:19.499785 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488782 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:37:19.499785 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488785 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:37:19.499785 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488788 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:37:19.499785 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488791 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:37:19.499785 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488794 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:37:19.499785 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488796 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:37:19.499785 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488799 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:37:19.499785 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488801 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:37:19.499785 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488804 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:37:19.499785 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488807 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:37:19.499785 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488809 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:37:19.499785 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488812 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:37:19.499785 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488815 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:37:19.499785 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488817 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:37:19.499785 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488820 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:37:19.499785 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488822 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:37:19.499785 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488825 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:37:19.499785 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488828 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:37:19.500319 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488830 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:37:19.500319 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488833 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:37:19.500319 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488835 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:37:19.500319 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488838 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:37:19.500319 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488841 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:37:19.500319 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488844 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:37:19.500319 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488847 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:37:19.500319 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488850 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:37:19.500319 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488853 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:37:19.500319 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488856 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:37:19.500319 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488871 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:37:19.500319 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488874 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:37:19.500319 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488876 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:37:19.500319 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488881 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:37:19.500319 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488883 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:37:19.500319 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488886 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:37:19.500319 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488889 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:37:19.500319 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488892 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:37:19.500319 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488895 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:37:19.500319 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488898 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:37:19.501094 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488901 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:37:19.501094 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488903 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:37:19.501094 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488906 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:37:19.501094 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488909 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:37:19.501094 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488911 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:37:19.501094 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488914 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:37:19.501094 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488917 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:37:19.501094 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488919 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:37:19.501094 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488922 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:37:19.501094 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488925 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:37:19.501094 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488928 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:37:19.501094 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488930 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:37:19.501094 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488933 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:37:19.501094 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488935 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:37:19.501094 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488938 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:37:19.501094 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488941 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:37:19.501094 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488943 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:37:19.501094 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488946 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:37:19.501094 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488949 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:37:19.501094 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488951 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:37:19.501664 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488954 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:37:19.501664 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488958 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:37:19.501664 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488961 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:37:19.501664 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488963 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:37:19.501664 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488968 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:37:19.501664 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488972 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:37:19.501664 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488975 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:37:19.501664 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488977 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:37:19.501664 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488980 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:37:19.501664 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488983 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:37:19.501664 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488986 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:37:19.501664 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488989 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:37:19.501664 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488994 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:37:19.501664 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.488997 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:37:19.501664 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.489000 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:37:19.501664 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.489002 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:37:19.501664 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.489005 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:37:19.501664 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.489007 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:37:19.501664 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.489010 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:37:19.502128 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.489012 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:37:19.502128 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.489015 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:37:19.502128 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.489018 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:37:19.502128 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.489021 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:37:19.502128 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.489023 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:37:19.502128 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.489026 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:37:19.502128 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.489030 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:37:19.502128 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.489032 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:37:19.502128 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.489035 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:37:19.502128 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.495551 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 20:37:19.502128 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.495566 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 20:37:19.502128 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.496663 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 20:37:19.502128 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.496670 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 20:37:19.502128 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496732 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:37:19.502128 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496737 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:37:19.502128 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496740 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:37:19.502128 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496743 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:37:19.502128 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496746 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:37:19.502128 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496749 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:37:19.502585 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496752 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:37:19.502585 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496754 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:37:19.502585 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496757 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:37:19.502585 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496759 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:37:19.502585 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496762 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:37:19.502585 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496765 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:37:19.502585 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496768 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:37:19.502585 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496771 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:37:19.502585 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496773 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:37:19.502585 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496776 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:37:19.502585 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496779 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:37:19.502585 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496782 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:37:19.502585 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496784 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:37:19.502585 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496787 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:37:19.502585 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496789 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:37:19.502585 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496792 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:37:19.502585 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496795 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:37:19.502585 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496797 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:37:19.502585 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496800 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:37:19.502585 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496803 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:37:19.503080 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496806 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:37:19.503080 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496808 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:37:19.503080 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496811 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:37:19.503080 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496815 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:37:19.503080 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496818 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:37:19.503080 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496821 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:37:19.503080 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496824 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:37:19.503080 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496827 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:37:19.503080 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496830 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:37:19.503080 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496833 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:37:19.503080 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496835 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:37:19.503080 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496838 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:37:19.503080 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496840 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:37:19.503080 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496843 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:37:19.503080 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496846 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:37:19.503080 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496850 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:37:19.503080 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496854 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:37:19.503080 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496870 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:37:19.503080 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496875 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:37:19.503530 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496878 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:37:19.503530 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496880 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:37:19.503530 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496883 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:37:19.503530 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496886 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:37:19.503530 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496889 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:37:19.503530 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496892 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:37:19.503530 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496894 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:37:19.503530 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496897 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:37:19.503530 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496899 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:37:19.503530 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496903 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:37:19.503530 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496907 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:37:19.503530 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496910 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:37:19.503530 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496912 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:37:19.503530 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496915 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:37:19.503530 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496918 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:37:19.503530 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496921 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:37:19.503530 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496924 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:37:19.503530 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496927 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:37:19.503530 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496930 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:37:19.504001 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496934 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:37:19.504001 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496936 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:37:19.504001 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496939 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:37:19.504001 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496942 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:37:19.504001 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496945 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:37:19.504001 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496947 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:37:19.504001 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496950 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:37:19.504001 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496952 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:37:19.504001 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496955 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:37:19.504001 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496957 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:37:19.504001 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496960 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:37:19.504001 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496962 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:37:19.504001 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496965 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:37:19.504001 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496968 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:37:19.504001 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496970 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:37:19.504001 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496973 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:37:19.504001 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496975 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:37:19.504001 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496978 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:37:19.504001 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496981 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:37:19.504001 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496983 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:37:19.504461 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496986 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:37:19.504461 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496988 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:37:19.504461 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496991 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:37:19.504461 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496993 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:37:19.504461 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496996 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:37:19.504461 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.496999 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:37:19.504461 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497001 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:37:19.504461 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497004 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:37:19.504461 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497007 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:37:19.504461 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497010 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:37:19.504461 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497013 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:37:19.504461 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497017 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:37:19.504461 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497019 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:37:19.504461 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497022 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:37:19.504461 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497025 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:37:19.504461 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497028 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:37:19.504461 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497031 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:37:19.504461 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497033 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:37:19.504461 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497036 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:37:19.504461 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497038 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:37:19.504954 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497041 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:37:19.504954 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497044 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:37:19.504954 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.497072 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:37:19.504954 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497282 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:37:19.504954 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497289 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:37:19.504954 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497293 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:37:19.504954 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497296 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:37:19.504954 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497299 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:37:19.504954 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497303 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:37:19.504954 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497306 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:37:19.504954 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497308 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:37:19.504954 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497311 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:37:19.504954 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497314 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:37:19.504954 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497316 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:37:19.504954 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497319 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:37:19.504954 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497322 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:37:19.505346 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497325 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:37:19.505346 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497327 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:37:19.505346 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497330 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:37:19.505346 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497333 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:37:19.505346 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497336 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:37:19.505346 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497338 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:37:19.505346 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497341 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:37:19.505346 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497344 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:37:19.505346 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497347 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:37:19.505346 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497350 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:37:19.505346 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497353 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:37:19.505346 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497356 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:37:19.505346 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497358 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:37:19.505346 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497361 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:37:19.505346 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497363 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:37:19.505346 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497366 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:37:19.505346 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497369 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:37:19.505346 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497372 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:37:19.505346 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497374 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:37:19.505803 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497377 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:37:19.505803 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497379 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:37:19.505803 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497382 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:37:19.505803 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497385 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:37:19.505803 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497388 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:37:19.505803 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497390 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:37:19.505803 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497393 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:37:19.505803 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497396 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:37:19.505803 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497398 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:37:19.505803 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497401 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:37:19.505803 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497403 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:37:19.505803 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497405 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:37:19.505803 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497408 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:37:19.505803 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497412 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:37:19.505803 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497416 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:37:19.505803 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497420 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:37:19.505803 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497424 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:37:19.505803 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497428 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:37:19.505803 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497431 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:37:19.506262 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497434 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:37:19.506262 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497437 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:37:19.506262 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497439 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:37:19.506262 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497448 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:37:19.506262 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497451 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:37:19.506262 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497454 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:37:19.506262 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497457 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:37:19.506262 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497460 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:37:19.506262 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497462 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:37:19.506262 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497465 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:37:19.506262 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497467 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:37:19.506262 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497470 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:37:19.506262 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497472 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:37:19.506262 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497476 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:37:19.506262 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497492 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:37:19.506262 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497495 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:37:19.506262 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497498 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:37:19.506262 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497501 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:37:19.506262 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497504 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:37:19.506262 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497507 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:37:19.506738 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497509 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:37:19.506738 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497512 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:37:19.506738 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497514 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:37:19.506738 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497517 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:37:19.506738 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497520 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:37:19.506738 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497522 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:37:19.506738 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497525 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:37:19.506738 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497528 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:37:19.506738 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497530 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:37:19.506738 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497533 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:37:19.506738 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497536 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:37:19.506738 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497538 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:37:19.506738 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497541 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:37:19.506738 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497543 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:37:19.506738 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497546 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:37:19.506738 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497549 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:37:19.506738 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497557 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:37:19.506738 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497560 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:37:19.506738 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497562 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:37:19.507195 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497565 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:37:19.507195 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497568 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:37:19.507195 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497570 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:37:19.507195 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497573 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:37:19.507195 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497575 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:37:19.507195 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497578 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:37:19.507195 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497581 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:37:19.507195 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497583 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:37:19.507195 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497586 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:37:19.507195 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497588 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:37:19.507195 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497591 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:37:19.507195 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497594 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:37:19.507195 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497597 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:37:19.507195 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497599 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:37:19.507195 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497601 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:37:19.507195 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497604 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:37:19.507565 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.497628 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:37:19.507565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497664 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:37:19.507565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497674 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:37:19.507565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497677 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:37:19.507565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497680 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:37:19.507565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497683 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:37:19.507565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497685 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:37:19.507565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497688 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:37:19.507565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497691 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:37:19.507565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497694 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:37:19.507565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497696 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:37:19.507565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497699 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:37:19.507565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497701 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:37:19.507565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497704 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:37:19.507565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497712 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:37:19.507565 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497715 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:37:19.507951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497718 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:37:19.507951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497720 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:37:19.507951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497723 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:37:19.507951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497725 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:37:19.507951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497728 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:37:19.507951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497731 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:37:19.507951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497733 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:37:19.507951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497736 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:37:19.507951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497738 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:37:19.507951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497741 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:37:19.507951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497744 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:37:19.507951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497746 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:37:19.507951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497749 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:37:19.507951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497751 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:37:19.507951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497754 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:37:19.507951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497757 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:37:19.507951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497759 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:37:19.507951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497761 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:37:19.507951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497764 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:37:19.507951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497767 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:37:19.508445 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497769 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:37:19.508445 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497772 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:37:19.508445 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497775 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:37:19.508445 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497778 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:37:19.508445 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497780 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:37:19.508445 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497783 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:37:19.508445 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497785 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:37:19.508445 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497788 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:37:19.508445 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497790 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:37:19.508445 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497793 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:37:19.508445 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497795 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:37:19.508445 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497803 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:37:19.508445 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497806 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:37:19.508445 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497809 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:37:19.508445 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497811 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:37:19.508445 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497814 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:37:19.508445 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497816 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:37:19.508445 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497819 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:37:19.508445 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497821 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:37:19.508445 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497824 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:37:19.508929 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497826 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:37:19.508929 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497829 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:37:19.508929 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497831 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:37:19.508929 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497834 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:37:19.508929 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497836 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:37:19.508929 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497839 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:37:19.508929 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497841 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:37:19.508929 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497844 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:37:19.508929 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497846 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:37:19.508929 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497849 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:37:19.508929 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497851 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:37:19.508929 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497854 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:37:19.508929 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497857 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:37:19.508929 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497875 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:37:19.508929 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497878 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:37:19.508929 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497881 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:37:19.508929 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497883 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:37:19.508929 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497886 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:37:19.508929 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497888 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:37:19.509403 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497891 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:37:19.509403 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497893 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:37:19.509403 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497896 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:37:19.509403 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497899 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:37:19.509403 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497901 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:37:19.509403 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497910 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:37:19.509403 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497913 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:37:19.509403 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497915 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:37:19.509403 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497918 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:37:19.509403 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497921 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:37:19.509403 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497924 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:37:19.509403 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497926 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:37:19.509403 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497929 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:37:19.509403 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497931 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:37:19.509403 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497935 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:37:19.509403 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497937 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:37:19.509403 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497940 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:37:19.509403 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497943 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:37:19.509403 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497945 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:37:19.509834 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497949 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:37:19.509834 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497954 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:37:19.509834 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497958 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:37:19.509834 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497961 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:37:19.509834 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497963 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:37:19.509834 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497966 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:37:19.509834 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497969 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:37:19.509834 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497972 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:37:19.509834 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497975 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:37:19.509834 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497977 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:37:19.509834 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497980 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:37:19.509834 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497983 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:37:19.509834 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:19.497986 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:37:19.509834 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.498044 2580 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 20:37:19.509834 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.500049 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 20:37:19.509834 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.501973 2580 server.go:1014] "Starting client certificate rotation" Oct 30 20:37:19.509834 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.502075 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 20:37:19.509834 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.502823 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 20:37:19.510382 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.504813 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:37:19.510382 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.507629 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:37:19.518476 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.518457 2580 log.go:25] "Validated CRI v1 runtime API" Oct 30 20:37:19.520815 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.520797 2580 log.go:25] "Validated CRI v1 image API" Oct 30 20:37:19.521736 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.521711 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 20:37:19.525019 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.524998 2580 fs.go:135] Filesystem UUIDs: map[1d655983-1288-4cda-9453-a48fd6e711c1:/dev/nvme0n1p4 32dcfa16-29c8-4c0f-b136-74d27f27f410:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 20:37:19.525092 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.525018 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 20:37:19.531594 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.531468 2580 manager.go:217] Machine: {Timestamp:2026-10-30 20:37:19.529382915 +0000 UTC m=+0.314536302 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101197 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26dece88547659e225b6460b365b83 SystemUUID:ec26dece-8854-7659-e225-b6460b365b83 BootID:c4f62b97-d062-418a-a4a7-44b6f49b9276 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c1:cd:46:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c1:cd:46:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:74:d1:5b:1a:99 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 20:37:19.531594 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.531586 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 20:37:19.531758 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.531740 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 20:37:19.534746 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.534721 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 20:37:19.534898 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.534746 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-52.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 20:37:19.534945 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.534900 2580 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 20:37:19.534945 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.534909 2580 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 20:37:19.535041 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.534955 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:37:19.535041 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.534970 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:37:19.535750 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.535740 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 20:37:19.539325 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.539313 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 20:37:19.539445 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.539436 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 20:37:19.539476 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.539470 2580 kubelet.go:493] "Attempting to sync node with API server" Oct 30 20:37:19.539506 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.539485 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 20:37:19.539506 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.539495 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 20:37:19.539506 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.539504 2580 kubelet.go:405] "Adding apiserver pod source" Oct 30 20:37:19.539628 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.539520 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 20:37:19.540071 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.540055 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:37:19.540376 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.540355 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 20:37:19.543628 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.543610 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 20:37:19.548661 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.548645 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:37:19.549384 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.549372 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 20:37:19.549431 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.549409 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 20:37:19.549559 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.549549 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 20:37:19.549588 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.549565 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 20:37:19.549588 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.549571 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 20:37:19.549588 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.549577 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 20:37:19.549588 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.549583 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 20:37:19.549588 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.549588 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 20:37:19.550177 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.550168 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 20:37:19.550208 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.550178 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 20:37:19.550208 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.550186 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 20:37:19.550208 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.550192 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 20:37:19.550208 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.550201 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 20:37:19.550208 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.550210 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 20:37:19.551310 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.551300 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 20:37:19.551348 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.551310 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 20:37:19.555279 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.555265 2580 server.go:1263] "Started kubelet" Oct 30 20:37:19.555390 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.555361 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 20:37:19.555445 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.555358 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 20:37:19.555445 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.555430 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 20:37:19.556123 ip-10-0-130-52 systemd[1]: Started Kubernetes Kubelet. Oct 30 20:37:19.557205 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.556434 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 20:37:19.558927 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.557467 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-52.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 20:37:19.559016 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.557482 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 20:37:19.559016 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.557502 2580 server.go:317] "Adding debug handlers to kubelet server" Oct 30 20:37:19.559016 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.557503 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-52.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 20:37:19.559724 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.559689 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 20:37:19.561572 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.561555 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 20:37:19.561668 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.561641 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 20:37:19.561668 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.561648 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:19.562260 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.562133 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 20:37:19.562260 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.562135 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 20:37:19.562260 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.562156 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 20:37:19.562440 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.562306 2580 reconstruct.go:97] "Volume reconstruction finished" Oct 30 20:37:19.562440 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.562315 2580 reconciler.go:29] "Reconciler: start to sync state" Oct 30 20:37:19.566319 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.563508 2580 factory.go:55] Registering systemd factory Oct 30 20:37:19.566319 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.563528 2580 factory.go:223] Registration of the systemd container factory successfully Oct 30 20:37:19.566319 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.563838 2580 factory.go:153] Registering CRI-O factory Oct 30 20:37:19.566319 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.563851 2580 factory.go:223] Registration of the crio container factory successfully Oct 30 20:37:19.566319 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.563937 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 20:37:19.566319 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.563955 2580 factory.go:103] Registering Raw factory Oct 30 20:37:19.566319 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.563969 2580 manager.go:1199] Started watching for new ooms in manager Oct 30 20:37:19.566319 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.563559 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a977d7686 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.55523751 +0000 UTC m=+0.340390891,LastTimestamp:2026-10-30 20:37:19.55523751 +0000 UTC m=+0.340390891,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.566319 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.564956 2580 manager.go:319] Starting recovery of all containers Oct 30 20:37:19.567033 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.567015 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-52.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 20:37:19.573382 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.573348 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 20:37:19.577220 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.577204 2580 manager.go:324] Recovery completed Oct 30 20:37:19.582017 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.582005 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:19.583813 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.583795 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:19.583881 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.583828 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:19.583881 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.583838 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:19.586009 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.585992 2580 cpu_manager.go:226] "Starting" policy="none" Oct 30 20:37:19.586009 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.586005 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 20:37:19.586154 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.586024 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 20:37:19.586154 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.586044 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:19.586154 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.586046 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a99318072 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583813746 +0000 UTC m=+0.368967126,LastTimestamp:2026-10-30 20:37:19.583813746 +0000 UTC m=+0.368967126,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.586751 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.586732 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:19.586833 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.586764 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:19.586833 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.586778 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:19.588295 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.588278 2580 policy_none.go:50] "Start" Oct 30 20:37:19.588362 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.588299 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 20:37:19.588362 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.588314 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 20:37:19.590440 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.590427 2580 policy_none.go:44] "Start" Oct 30 20:37:19.591873 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.591839 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cj5hx" Oct 30 20:37:19.599329 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.599264 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931cb07 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583832839 +0000 UTC m=+0.368986220,LastTimestamp:2026-10-30 20:37:19.583832839 +0000 UTC m=+0.368986220,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.609046 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.608982 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931ef03 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583842051 +0000 UTC m=+0.368995431,LastTimestamp:2026-10-30 20:37:19.583842051 +0000 UTC m=+0.368995431,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.616491 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.616462 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 20:37:19.619640 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.619545 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-52.us-east-2.compute.internal.18e3691a99318072\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a99318072 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583813746 +0000 UTC m=+0.368967126,LastTimestamp:2026-10-30 20:37:19.58674855 +0000 UTC m=+0.371901935,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.621630 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.621615 2580 manager.go:341] "Starting Device Plugin manager" Oct 30 20:37:19.621719 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.621656 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 20:37:19.621719 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.621667 2580 server.go:87] "Starting device plugin registration server" Oct 30 20:37:19.621947 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.621934 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 20:37:19.622014 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.621948 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 20:37:19.622061 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.622021 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 20:37:19.622621 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.622194 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 20:37:19.622621 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.622207 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 20:37:19.622621 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.622527 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 20:37:19.622621 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.622605 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:19.628744 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.628681 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931cb07\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931cb07 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583832839 +0000 UTC m=+0.368986220,LastTimestamp:2026-10-30 20:37:19.586771359 +0000 UTC m=+0.371924742,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.632661 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.632604 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931ef03\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931ef03 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583842051 +0000 UTC m=+0.368995431,LastTimestamp:2026-10-30 20:37:19.586783906 +0000 UTC m=+0.371937287,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.638805 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.638728 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a9b71e243 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated limits on system reserved cgroup /system.slice,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.621587523 +0000 UTC m=+0.406740907,LastTimestamp:2026-10-30 20:37:19.621587523 +0000 UTC m=+0.406740907,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.660645 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.644629 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a9b8ca911 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.623342353 +0000 UTC m=+0.408495725,LastTimestamp:2026-10-30 20:37:19.623342353 +0000 UTC m=+0.408495725,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.694212 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.694173 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 20:37:19.695335 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.695315 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 20:37:19.695421 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.695344 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 20:37:19.695421 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.695367 2580 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 20:37:19.695421 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.695407 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 20:37:19.707669 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.707648 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 20:37:19.722770 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.722752 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:19.723581 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.723565 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:19.723679 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.723599 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:19.723679 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.723616 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:19.723679 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.723647 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:19.734518 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.734372 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-52.us-east-2.compute.internal.18e3691a99318072\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a99318072 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583813746 +0000 UTC m=+0.368967126,LastTimestamp:2026-10-30 20:37:19.7235812 +0000 UTC m=+0.508734589,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.742506 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.742486 2580 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-130-52.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:19.742666 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.742598 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931cb07\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931cb07 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583832839 +0000 UTC m=+0.368986220,LastTimestamp:2026-10-30 20:37:19.723606361 +0000 UTC m=+0.508759749,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.750916 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.750834 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931ef03\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931ef03 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583842051 +0000 UTC m=+0.368995431,LastTimestamp:2026-10-30 20:37:19.723622342 +0000 UTC m=+0.508775725,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.776504 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.776479 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-52.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms" Oct 30 20:37:19.795700 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.795673 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal"] Oct 30 20:37:19.795748 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.795733 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:19.796567 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.796552 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:19.796623 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.796582 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:19.796623 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.796594 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:19.798841 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.798827 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:19.798988 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.798973 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:19.799032 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.799005 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:19.799551 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.799536 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:19.799641 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.799563 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:19.799641 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.799579 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:19.799641 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.799541 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:19.799789 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.799649 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:19.799789 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.799664 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:19.801740 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.801727 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:19.801786 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.801750 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:19.802957 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.802939 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:19.803033 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.802965 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:19.803033 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.802978 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:19.805247 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.805158 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-52.us-east-2.compute.internal.18e3691a99318072\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a99318072 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583813746 +0000 UTC m=+0.368967126,LastTimestamp:2026-10-30 20:37:19.796568205 +0000 UTC m=+0.581721589,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.813602 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.813541 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931cb07\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931cb07 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583832839 +0000 UTC m=+0.368986220,LastTimestamp:2026-10-30 20:37:19.796587288 +0000 UTC m=+0.581740671,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.824126 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.824058 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931ef03\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931ef03 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583842051 +0000 UTC m=+0.368995431,LastTimestamp:2026-10-30 20:37:19.796597543 +0000 UTC m=+0.581750927,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.832382 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.832314 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-52.us-east-2.compute.internal.18e3691a99318072\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a99318072 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583813746 +0000 UTC m=+0.368967126,LastTimestamp:2026-10-30 20:37:19.79955136 +0000 UTC m=+0.584704748,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.835531 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.835513 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:19.838391 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.838372 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:19.842214 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.842147 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931cb07\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931cb07 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583832839 +0000 UTC m=+0.368986220,LastTimestamp:2026-10-30 20:37:19.799570085 +0000 UTC m=+0.584723471,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.854130 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.854063 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931ef03\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931ef03 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583842051 +0000 UTC m=+0.368995431,LastTimestamp:2026-10-30 20:37:19.799585378 +0000 UTC m=+0.584738765,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.862258 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.862195 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-52.us-east-2.compute.internal.18e3691a99318072\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a99318072 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583813746 +0000 UTC m=+0.368967126,LastTimestamp:2026-10-30 20:37:19.799616937 +0000 UTC m=+0.584770318,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.863919 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.863900 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0459ad6c6db5719be39646515856f6a3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal\" (UID: \"0459ad6c6db5719be39646515856f6a3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:19.863998 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.863925 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0459ad6c6db5719be39646515856f6a3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal\" (UID: \"0459ad6c6db5719be39646515856f6a3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:19.863998 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.863942 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/879a456e73ba77e5a50d63568935393e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal\" (UID: \"879a456e73ba77e5a50d63568935393e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:19.870365 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.870308 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931cb07\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931cb07 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583832839 +0000 UTC m=+0.368986220,LastTimestamp:2026-10-30 20:37:19.799657695 +0000 UTC m=+0.584811076,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.880686 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.880624 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931ef03\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931ef03 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583842051 +0000 UTC m=+0.368995431,LastTimestamp:2026-10-30 20:37:19.799668134 +0000 UTC m=+0.584821515,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.893985 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.893928 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-52.us-east-2.compute.internal.18e3691a99318072\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a99318072 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583813746 +0000 UTC m=+0.368967126,LastTimestamp:2026-10-30 20:37:19.802952265 +0000 UTC m=+0.588105646,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.905184 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.905118 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931cb07\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931cb07 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583832839 +0000 UTC m=+0.368986220,LastTimestamp:2026-10-30 20:37:19.802971288 +0000 UTC m=+0.588124677,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.914977 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.914918 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931ef03\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a9931ef03 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583842051 +0000 UTC m=+0.368995431,LastTimestamp:2026-10-30 20:37:19.802984234 +0000 UTC m=+0.588137619,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.943067 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.943048 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:19.943894 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.943874 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:19.943987 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.943904 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:19.943987 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.943914 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:19.943987 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.943942 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:19.952716 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.952651 2580 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-130-52.us-east-2.compute.internal.18e3691a99318072\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-52.us-east-2.compute.internal.18e3691a99318072 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-52.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-52.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:19.583813746 +0000 UTC m=+0.368967126,LastTimestamp:2026-10-30 20:37:19.943890108 +0000 UTC m=+0.729043488,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:19.954632 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:19.954613 2580 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-130-52.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:19.964942 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.964920 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0459ad6c6db5719be39646515856f6a3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal\" (UID: \"0459ad6c6db5719be39646515856f6a3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:19.965019 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.964947 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0459ad6c6db5719be39646515856f6a3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal\" (UID: \"0459ad6c6db5719be39646515856f6a3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:19.965019 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.964964 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/879a456e73ba77e5a50d63568935393e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal\" (UID: \"879a456e73ba77e5a50d63568935393e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:19.965019 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.965014 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/879a456e73ba77e5a50d63568935393e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal\" (UID: \"879a456e73ba77e5a50d63568935393e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:19.965126 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.965033 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0459ad6c6db5719be39646515856f6a3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal\" (UID: \"0459ad6c6db5719be39646515856f6a3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:19.965126 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:19.965070 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0459ad6c6db5719be39646515856f6a3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal\" (UID: \"0459ad6c6db5719be39646515856f6a3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:20.138723 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:20.138649 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:20.142038 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:20.142022 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:20.185426 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:20.185271 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-52.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="800ms" Oct 30 20:37:20.354752 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:20.354718 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:20.355781 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:20.355761 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:20.355918 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:20.355800 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:20.355918 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:20.355813 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:20.355918 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:20.355841 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:20.374233 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:20.374207 2580 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-130-52.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:22.282184 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:22.282157 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-52.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 20:37:22.345948 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:22.345919 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-52.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 20:37:22.419216 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:22.419176 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:22.420476 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:22.420460 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:22.420553 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:22.420490 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:22.420553 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:22.420500 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:22.589716 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:22.589228 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879a456e73ba77e5a50d63568935393e.slice/crio-b4fc79cde41391c8690bb276a888b958b0fb62293a146794e3ec73f90451e3f4 WatchSource:0}: Error finding container b4fc79cde41391c8690bb276a888b958b0fb62293a146794e3ec73f90451e3f4: Status 404 returned error can't find the container with id b4fc79cde41391c8690bb276a888b958b0fb62293a146794e3ec73f90451e3f4 Oct 30 20:37:22.589929 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:22.589909 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0459ad6c6db5719be39646515856f6a3.slice/crio-6699dc1c4ebf7b70abd16bb5b79b34127d1801b261cf3b6e50ad3abbaee3b1db WatchSource:0}: Error finding container 6699dc1c4ebf7b70abd16bb5b79b34127d1801b261cf3b6e50ad3abbaee3b1db: Status 404 returned error can't find the container with id 6699dc1c4ebf7b70abd16bb5b79b34127d1801b261cf3b6e50ad3abbaee3b1db Oct 30 20:37:22.593450 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:22.593430 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:37:22.597273 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:22.597246 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 20:37:22.597345 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:22.597236 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal.18e3691b4c9d02ed kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal,UID:879a456e73ba77e5a50d63568935393e,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e\",Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:22.593981165 +0000 UTC m=+1.601229467,LastTimestamp:2026-10-30 20:37:22.593981165 +0000 UTC m=+1.601229467,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:22.602288 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:22.602225 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal.18e3691b4caf3d06 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal,UID:0459ad6c6db5719be39646515856f6a3,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\",Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:22.595175686 +0000 UTC m=+1.602423978,LastTimestamp:2026-10-30 20:37:22.595175686 +0000 UTC m=+1.602423978,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:22.719918 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:22.719891 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 20:37:22.773366 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:22.773340 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-52.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="1.6s" Oct 30 20:37:22.781215 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:22.781193 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 20:37:22.952410 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:22.952335 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:22.954162 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:22.954144 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:22.954239 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:22.954176 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:22.954239 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:22.954186 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:22.954239 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:22.954212 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:22.972760 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:22.972733 2580 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-130-52.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:23.352115 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:23.352039 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-52.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 20:37:23.479321 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:23.479258 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal" event={"ID":"879a456e73ba77e5a50d63568935393e","Type":"ContainerStarted","Data":"b4fc79cde41391c8690bb276a888b958b0fb62293a146794e3ec73f90451e3f4"} Oct 30 20:37:23.480803 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:23.480757 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" event={"ID":"0459ad6c6db5719be39646515856f6a3","Type":"ContainerStarted","Data":"6699dc1c4ebf7b70abd16bb5b79b34127d1801b261cf3b6e50ad3abbaee3b1db"} Oct 30 20:37:24.040277 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:24.040197 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal.18e3691ba23a59cb openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal,UID:0459ad6c6db5719be39646515856f6a3,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\" in 1.435s (1.435s including waiting). Image size: 272339155 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:24.030355915 +0000 UTC m=+3.037604223,LastTimestamp:2026-10-30 20:37:24.030355915 +0000 UTC m=+3.037604223,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:24.165305 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:24.165177 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal.18e3691ba9a5dac1 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal,UID:879a456e73ba77e5a50d63568935393e,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e\" in 1.56s (1.56s including waiting). Image size: 294084393 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:24.154841793 +0000 UTC m=+3.162090106,LastTimestamp:2026-10-30 20:37:24.154841793 +0000 UTC m=+3.162090106,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:24.228160 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:24.227817 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal.18e3691bad6ee39a kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal,UID:879a456e73ba77e5a50d63568935393e,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Created,Message:Container created,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:24.218348442 +0000 UTC m=+3.225596730,LastTimestamp:2026-10-30 20:37:24.218348442 +0000 UTC m=+3.225596730,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:24.237911 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:24.237833 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal.18e3691bade26ebd kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal,UID:879a456e73ba77e5a50d63568935393e,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Started,Message:Container started,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:24.225920701 +0000 UTC m=+3.233169005,LastTimestamp:2026-10-30 20:37:24.225920701 +0000 UTC m=+3.233169005,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:24.248811 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:24.248721 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal.18e3691bae03d9e6 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal,UID:0459ad6c6db5719be39646515856f6a3,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Container created,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:24.228110822 +0000 UTC m=+3.235359132,LastTimestamp:2026-10-30 20:37:24.228110822 +0000 UTC m=+3.235359132,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:24.256840 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:24.256747 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal.18e3691bae7e1fae openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal,UID:0459ad6c6db5719be39646515856f6a3,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Container started,Source:EventSource{Component:kubelet,Host:ip-10-0-130-52.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:37:24.236124078 +0000 UTC m=+3.243372368,LastTimestamp:2026-10-30 20:37:24.236124078 +0000 UTC m=+3.243372368,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-52.us-east-2.compute.internal,}" Oct 30 20:37:24.347110 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:24.347044 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-52.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 20:37:24.382803 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:24.382760 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-52.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="3.2s" Oct 30 20:37:24.472080 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:24.472051 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 20:37:24.484016 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:24.483989 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal" event={"ID":"879a456e73ba77e5a50d63568935393e","Type":"ContainerStarted","Data":"f00287dbbc4430696cba50bbf839e2b51a1add1a7e83fa8ba033f5dbc6865aa3"} Oct 30 20:37:24.484016 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:24.484010 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:24.485040 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:24.485022 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:24.485136 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:24.485059 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:24.485136 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:24.485075 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:24.485290 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:24.485269 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:24.485532 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:24.485515 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" event={"ID":"0459ad6c6db5719be39646515856f6a3","Type":"ContainerStarted","Data":"e73fb87d6e8625c463b8e413b77079eebbf77689c5517adebe0f692312539cc5"} Oct 30 20:37:24.485581 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:24.485560 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:24.486211 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:24.486197 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:24.486300 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:24.486221 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:24.486300 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:24.486234 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:24.486409 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:24.486381 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:24.518614 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:24.518580 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 20:37:24.573132 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:24.573103 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:24.574105 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:24.574089 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:24.574200 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:24.574123 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:24.574200 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:24.574139 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:24.574200 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:24.574175 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:24.592801 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:24.592776 2580 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-130-52.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:24.645209 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:24.645142 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:24.646166 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:24.646148 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:24.646242 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:24.646181 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:24.646242 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:24.646195 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:25.202989 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:25.202960 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-52.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 20:37:25.346052 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:25.346024 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-52.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 20:37:25.487613 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:25.487555 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:25.488000 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:25.487561 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:25.488447 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:25.488425 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:25.488559 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:25.488460 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:25.488559 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:25.488472 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:25.488559 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:25.488427 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:25.488559 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:25.488506 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:25.488559 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:25.488518 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:25.488731 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:25.488716 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:25.488878 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:25.488866 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:25.673379 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:25.673352 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 20:37:26.350194 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:26.350168 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-52.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 20:37:27.348216 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:27.348189 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-52.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 20:37:27.591662 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:27.591630 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-52.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="6.4s" Oct 30 20:37:27.793781 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:27.793687 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:27.794665 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:27.794648 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:27.794728 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:27.794681 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:27.794728 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:27.794698 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:27.794815 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:27.794731 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:27.814925 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:27.814898 2580 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-130-52.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:28.346340 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:28.346306 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-52.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 20:37:28.884817 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:28.884786 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-52.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 20:37:29.346232 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:29.346157 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-52.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 20:37:29.407552 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:29.407518 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 20:37:29.447806 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:29.447763 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:29.448653 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:29.448633 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:29.448750 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:29.448668 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:29.448750 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:29.448685 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:29.595910 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:29.595880 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 30 20:37:29.791553 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:29.791484 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cj5hx" Oct 30 20:37:30.279362 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:30.279290 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 20:37:30.349994 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:30.349971 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-130-52.us-east-2.compute.internal" not found Oct 30 20:37:30.367707 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:30.367690 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-130-52.us-east-2.compute.internal" not found Oct 30 20:37:30.426120 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:30.426103 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-130-52.us-east-2.compute.internal" not found Oct 30 20:37:30.465203 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:30.465183 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:37:30.704253 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:30.704178 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-130-52.us-east-2.compute.internal" not found Oct 30 20:37:30.704253 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:30.704205 2580 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-130-52.us-east-2.compute.internal" not found Oct 30 20:37:30.739597 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:30.739568 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-130-52.us-east-2.compute.internal" not found Oct 30 20:37:30.753715 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:30.753694 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-130-52.us-east-2.compute.internal" not found Oct 30 20:37:30.797147 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:30.797107 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:32:29 +0000 UTC" deadline="2028-04-09 06:19:55.898394192 +0000 UTC" Oct 30 20:37:30.797147 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:30.797143 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12633h42m25.101253942s" Oct 30 20:37:30.809081 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:30.809059 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-130-52.us-east-2.compute.internal" not found Oct 30 20:37:31.069027 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:31.068971 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-130-52.us-east-2.compute.internal" not found Oct 30 20:37:31.069027 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:31.068996 2580 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-130-52.us-east-2.compute.internal" not found Oct 30 20:37:31.326718 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:31.326636 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-130-52.us-east-2.compute.internal" not found Oct 30 20:37:31.342850 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:31.342827 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-130-52.us-east-2.compute.internal" not found Oct 30 20:37:31.401386 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:31.401365 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-130-52.us-east-2.compute.internal" not found Oct 30 20:37:31.401496 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:31.401426 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:31.679879 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:31.679795 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-130-52.us-east-2.compute.internal" not found Oct 30 20:37:31.679879 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:31.679821 2580 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-130-52.us-east-2.compute.internal" not found Oct 30 20:37:33.702833 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:33.702804 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-130-52.us-east-2.compute.internal" not found Oct 30 20:37:33.717886 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:33.717855 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-130-52.us-east-2.compute.internal" not found Oct 30 20:37:33.774119 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:33.774092 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-130-52.us-east-2.compute.internal" not found Oct 30 20:37:33.998408 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:33.998339 2580 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-52.us-east-2.compute.internal\" not found" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:34.047429 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:34.047404 2580 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-130-52.us-east-2.compute.internal" not found Oct 30 20:37:34.047429 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:34.047429 2580 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-130-52.us-east-2.compute.internal" not found Oct 30 20:37:34.214994 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:34.214957 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:34.216057 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:34.216039 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:34.216172 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:34.216073 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:34.216172 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:34.216087 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:34.216172 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:34.216122 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:34.226050 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:34.226026 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:34.226171 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:34.226063 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-52.us-east-2.compute.internal\": node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:34.245511 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:34.245485 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:34.346415 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:34.346316 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:34.447328 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:34.447288 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:34.547919 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:34.547874 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:34.648426 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:34.648345 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:34.748856 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:34.748819 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:34.849643 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:34.849607 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:34.950171 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:34.950102 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:35.050596 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:35.050565 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:35.151067 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:35.151031 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:35.251580 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:35.251499 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:35.352547 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:35.352504 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:35.453088 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:35.453059 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:35.505032 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:35.503429 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal_0459ad6c6db5719be39646515856f6a3/setup/0.log" Oct 30 20:37:35.505032 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:35.503493 2580 generic.go:356] "Generic (PLEG): container finished" podID="0459ad6c6db5719be39646515856f6a3" containerID="e73fb87d6e8625c463b8e413b77079eebbf77689c5517adebe0f692312539cc5" exitCode=1 Oct 30 20:37:35.505032 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:35.503530 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" event={"ID":"0459ad6c6db5719be39646515856f6a3","Type":"ContainerDied","Data":"e73fb87d6e8625c463b8e413b77079eebbf77689c5517adebe0f692312539cc5"} Oct 30 20:37:35.505032 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:35.503717 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:35.505786 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:35.505751 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:35.505844 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:35.505801 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:35.505844 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:35.505814 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:35.506005 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:35.505993 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:35.553712 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:35.553571 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:35.653997 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:35.653961 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:35.754462 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:35.754426 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:35.854866 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:35.854777 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:35.955677 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:35.955649 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:36.055993 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:36.055960 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:36.156509 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:36.156444 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:36.257000 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:36.256961 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:36.358062 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:36.358027 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:36.458345 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:36.458256 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:36.507590 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:36.507566 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal_0459ad6c6db5719be39646515856f6a3/setup/0.log" Oct 30 20:37:36.507715 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:36.507615 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" event={"ID":"0459ad6c6db5719be39646515856f6a3","Type":"ContainerStarted","Data":"86c55d4aea6a923581be42279421c6a7d7627e21c4599cc522d38f8b2bee8cda"} Oct 30 20:37:36.507811 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:36.507747 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:37:36.513089 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:36.513066 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:37:36.513202 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:36.513098 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:37:36.513202 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:36.513113 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:37:36.513355 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:36.513341 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:36.558836 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:36.558803 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:37:36.564343 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:36.564321 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:37:36.640577 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:36.640532 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:36.651010 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:36.650984 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:37:36.652448 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:36.652430 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:37:36.663551 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:36.663529 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:37:37.327334 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.327296 2580 apiserver.go:52] "Watching apiserver" Oct 30 20:37:37.347995 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.347968 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 20:37:37.359503 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.359485 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 20:37:37.421894 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.421867 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8sjrd" Oct 30 20:37:37.432046 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.432020 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8sjrd" Oct 30 20:37:37.432433 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.432416 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 20:37:37.432695 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.432679 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-554jb","openshift-multus/network-metrics-daemon-48zsl","openshift-ovn-kubernetes/ovnkube-node-rz98l","openshift-dns/node-resolver-vcndb","openshift-multus/multus-2kvw6","openshift-network-diagnostics/network-check-target-qs2dh","kube-system/global-pull-secret-syncer-5wp5n","openshift-network-operator/iptables-alerter-wzxhr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg","openshift-cluster-node-tuning-operator/tuned-wr47v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-n7lg7","kube-system/konnectivity-agent-grp6n","kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal"] Oct 30 20:37:37.436303 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.436286 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:37.436391 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:37.436365 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wp5n" podUID="a99706d1-d012-4dd9-b46a-5ec139287351" Oct 30 20:37:37.438963 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.438949 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-grp6n" Oct 30 20:37:37.441630 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.441613 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.444092 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.444073 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vcndb" Oct 30 20:37:37.446692 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.446677 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.449047 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.449032 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l4rxj\"" Oct 30 20:37:37.449135 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.449047 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x97dd\"" Oct 30 20:37:37.449278 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.449261 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:37:37.449339 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:37.449318 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs2dh" podUID="0832230b-2d6a-40f2-8d5f-dc1da218c7b8" Oct 30 20:37:37.449499 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.449482 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7lq4m\"" Oct 30 20:37:37.449629 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.449609 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 20:37:37.449719 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.449642 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 20:37:37.449719 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.449711 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 20:37:37.449859 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.449828 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 20:37:37.449859 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.449855 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 20:37:37.450008 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.449864 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 20:37:37.450008 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.449876 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 20:37:37.450008 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.449888 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 20:37:37.450008 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.449964 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 20:37:37.450147 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.450122 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 20:37:37.452035 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.452001 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 20:37:37.452527 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.452510 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.454555 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.454538 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-554jb" Oct 30 20:37:37.456030 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.456016 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 20:37:37.456441 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.456426 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 20:37:37.456576 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.456563 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 20:37:37.457356 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.457336 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:37.457422 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:37.457402 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48zsl" podUID="51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249" Oct 30 20:37:37.460903 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.460858 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.463390 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.463374 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wzxhr" Oct 30 20:37:37.464172 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.464153 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 20:37:37.464270 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.464232 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 20:37:37.464325 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.464276 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 20:37:37.464390 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.464369 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 20:37:37.464425 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.464408 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 20:37:37.466398 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.466383 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.470366 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.470345 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 20:37:37.470452 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.470421 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bcgjk\"" Oct 30 20:37:37.470512 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.470477 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-6lk59\"" Oct 30 20:37:37.470566 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.470532 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fxgqk\"" Oct 30 20:37:37.470613 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.470568 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-82bc7\"" Oct 30 20:37:37.470613 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.470605 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2hx48\"" Oct 30 20:37:37.471937 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.471919 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-chh5h\"" Oct 30 20:37:37.524327 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-ovn-node-metrics-cert\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.524436 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524383 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:37.524436 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524413 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c57ecae-89cb-4901-8f9e-e4586771f5d8-konnectivity-ca\") pod \"konnectivity-agent-grp6n\" (UID: \"4c57ecae-89cb-4901-8f9e-e4586771f5d8\") " pod="kube-system/konnectivity-agent-grp6n" Oct 30 20:37:37.524521 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524438 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-kubelet\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.524521 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524489 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-var-lib-openvswitch\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.524577 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524526 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-run-ovn\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.524577 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524543 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-log-socket\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.524634 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524579 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c57ecae-89cb-4901-8f9e-e4586771f5d8-agent-certs\") pod \"konnectivity-agent-grp6n\" (UID: \"4c57ecae-89cb-4901-8f9e-e4586771f5d8\") " pod="kube-system/konnectivity-agent-grp6n" Oct 30 20:37:37.524634 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524623 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-run-systemd\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.524689 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524651 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-run-ovn-kubernetes\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.524689 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524669 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-cni-bin\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.524745 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524696 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-cni-netd\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.524745 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524721 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.524823 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524744 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-ovnkube-config\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.524823 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524776 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-slash\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.524823 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524814 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-env-overrides\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.524910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524834 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-ovnkube-script-lib\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.524910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524884 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7662k\" (UniqueName: \"kubernetes.io/projected/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-kube-api-access-7662k\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.524910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524899 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a99706d1-d012-4dd9-b46a-5ec139287351-kubelet-config\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:37.524989 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524913 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a99706d1-d012-4dd9-b46a-5ec139287351-dbus\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:37.524989 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524927 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-systemd-units\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.524989 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524946 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-run-netns\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.524989 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.524972 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-etc-openvswitch\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.525099 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.525013 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-run-openvswitch\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.525099 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.525039 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-node-log\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.534924 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.534907 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 20:37:37.535148 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.535135 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 20:37:37.535193 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.535180 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 20:37:37.535234 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.535193 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:37:37.535265 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.535235 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 20:37:37.535365 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.535352 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:37:37.535425 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.535382 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 20:37:37.540901 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.540881 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 20:37:37.562969 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.562948 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bqhgw"] Oct 30 20:37:37.565691 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.565677 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.578791 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.578730 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:37:37.582606 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.582590 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 20:37:37.582606 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.582603 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 20:37:37.582714 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.582615 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5c74x\"" Oct 30 20:37:37.619423 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.619402 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 20:37:37.620641 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.620618 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 20:37:37.620723 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.620661 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 20:37:37.620884 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.620868 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 20:37:37.625208 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625185 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a99706d1-d012-4dd9-b46a-5ec139287351-dbus\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:37.625288 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625224 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ce76c71-8b23-49b4-a61c-9b14adea86c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.625288 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625250 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-var-lib-kubelet\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.625376 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625333 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:37.625376 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-var-lib-openvswitch\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.625456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625376 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a99706d1-d012-4dd9-b46a-5ec139287351-dbus\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:37.625456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625388 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-log-socket\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.625456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625412 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-run-k8s-cni-cncf-io\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.625456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625429 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-run-systemd\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.625456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625444 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-var-lib-openvswitch\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.625669 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625457 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-log-socket\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.625669 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:37.625467 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:37.625669 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625474 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-cni-netd\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.625669 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-cni-netd\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.625669 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625494 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-run-systemd\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.625669 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625513 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-run\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.625669 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:37.625542 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret podName:a99706d1-d012-4dd9-b46a-5ec139287351 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:38.125520598 +0000 UTC m=+17.132768908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret") pod "global-pull-secret-syncer-5wp5n" (UID: "a99706d1-d012-4dd9-b46a-5ec139287351") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:37.625669 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625579 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7krsp\" (UniqueName: \"kubernetes.io/projected/ab625ab1-de4f-4418-bd65-edb628d963fe-kube-api-access-7krsp\") pod \"node-resolver-vcndb\" (UID: \"ab625ab1-de4f-4418-bd65-edb628d963fe\") " pod="openshift-dns/node-resolver-vcndb" Oct 30 20:37:37.625669 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625604 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da9ceed2-3ab8-43dc-88a3-f20d7351ac30-host\") pod \"node-ca-554jb\" (UID: \"da9ceed2-3ab8-43dc-88a3-f20d7351ac30\") " pod="openshift-image-registry/node-ca-554jb" Oct 30 20:37:37.625669 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625628 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-run-multus-certs\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.625999 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-env-overrides\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.625999 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625707 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-sys\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.625999 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625759 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-lib-modules\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.625999 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625838 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ce76c71-8b23-49b4-a61c-9b14adea86c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.625999 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625879 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-socket-dir-parent\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.625999 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625908 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-run-openvswitch\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.625999 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625932 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b91b1fa-0886-4a2c-8005-bd6f77666874-host-slash\") pod \"iptables-alerter-wzxhr\" (UID: \"5b91b1fa-0886-4a2c-8005-bd6f77666874\") " pod="openshift-network-operator/iptables-alerter-wzxhr" Oct 30 20:37:37.625999 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625949 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-run-ovn-kubernetes\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.625999 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625966 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.625999 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625985 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-run-openvswitch\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.625999 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.625988 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-socket-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.626341 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626017 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.626341 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626046 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j47v7\" (UniqueName: \"kubernetes.io/projected/da9ceed2-3ab8-43dc-88a3-f20d7351ac30-kube-api-access-j47v7\") pod \"node-ca-554jb\" (UID: \"da9ceed2-3ab8-43dc-88a3-f20d7351ac30\") " pod="openshift-image-registry/node-ca-554jb" Oct 30 20:37:37.626341 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626063 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs\") pod \"network-metrics-daemon-48zsl\" (UID: \"51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249\") " pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:37.626341 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626087 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v69k5\" (UniqueName: \"kubernetes.io/projected/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-kube-api-access-v69k5\") pod \"network-metrics-daemon-48zsl\" (UID: \"51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249\") " pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:37.626341 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626135 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-env-overrides\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.626341 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626162 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-ovnkube-script-lib\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.626341 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626182 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mcpr\" (UniqueName: \"kubernetes.io/projected/6ce76c71-8b23-49b4-a61c-9b14adea86c7-kube-api-access-5mcpr\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.626341 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626161 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-run-ovn-kubernetes\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.626341 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626201 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-var-lib-cni-bin\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.626341 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626223 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-var-lib-kubelet\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.626341 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626315 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-daemon-config\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.626674 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626349 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-sysctl-d\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.626674 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626381 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-sysctl-conf\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.626674 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626423 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bsb9\" (UniqueName: \"kubernetes.io/projected/313047e7-bc02-42eb-933b-ab52575af3f7-kube-api-access-9bsb9\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.626674 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626459 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ce76c71-8b23-49b4-a61c-9b14adea86c7-system-cni-dir\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.626674 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626491 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-run-netns\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.626674 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626525 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-run-netns\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.626674 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626571 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-sysconfig\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.626674 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626598 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-registration-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.626674 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626621 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-device-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.626674 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626646 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da9ceed2-3ab8-43dc-88a3-f20d7351ac30-serviceca\") pod \"node-ca-554jb\" (UID: \"da9ceed2-3ab8-43dc-88a3-f20d7351ac30\") " pod="openshift-image-registry/node-ca-554jb" Oct 30 20:37:37.626674 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626649 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-ovnkube-script-lib\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.626674 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626671 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-system-cni-dir\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.627040 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626685 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-hostroot\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.627040 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626716 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9390b293-dd28-4713-8452-b9fb5d7ccaec-tmp\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.627040 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626742 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.627040 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626760 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-cni-bin\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.627040 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626810 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-cni-bin\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.627040 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626840 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c57ecae-89cb-4901-8f9e-e4586771f5d8-konnectivity-ca\") pod \"konnectivity-agent-grp6n\" (UID: \"4c57ecae-89cb-4901-8f9e-e4586771f5d8\") " pod="kube-system/konnectivity-agent-grp6n" Oct 30 20:37:37.627040 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626867 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-kubelet\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.627040 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-run-ovn\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.627040 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626906 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-kubelet\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.627040 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626922 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7456\" (UniqueName: \"kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456\") pod \"network-check-target-qs2dh\" (UID: \"0832230b-2d6a-40f2-8d5f-dc1da218c7b8\") " pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:37:37.627040 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626935 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-cnibin\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.627040 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626956 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-ovnkube-config\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.627040 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626970 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-host\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.627040 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.626984 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab625ab1-de4f-4418-bd65-edb628d963fe-hosts-file\") pod \"node-resolver-vcndb\" (UID: \"ab625ab1-de4f-4418-bd65-edb628d963fe\") " pod="openshift-dns/node-resolver-vcndb" Oct 30 20:37:37.627040 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627010 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ce76c71-8b23-49b4-a61c-9b14adea86c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.627040 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627038 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-run-ovn\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.627600 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627069 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ce76c71-8b23-49b4-a61c-9b14adea86c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.627600 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7662k\" (UniqueName: \"kubernetes.io/projected/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-kube-api-access-7662k\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.627600 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627131 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-modprobe-d\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.627600 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627168 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-systemd\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.627600 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627202 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b91b1fa-0886-4a2c-8005-bd6f77666874-iptables-alerter-script\") pod \"iptables-alerter-wzxhr\" (UID: \"5b91b1fa-0886-4a2c-8005-bd6f77666874\") " pod="openshift-network-operator/iptables-alerter-wzxhr" Oct 30 20:37:37.627600 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627231 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-run-netns\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.627600 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-etc-kubernetes\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.627600 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627270 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c57ecae-89cb-4901-8f9e-e4586771f5d8-konnectivity-ca\") pod \"konnectivity-agent-grp6n\" (UID: \"4c57ecae-89cb-4901-8f9e-e4586771f5d8\") " pod="kube-system/konnectivity-agent-grp6n" Oct 30 20:37:37.627600 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627280 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbdm6\" (UniqueName: \"kubernetes.io/projected/0137d1fc-8fa3-4b0e-8821-4776913088f7-kube-api-access-rbdm6\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.627600 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627315 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a99706d1-d012-4dd9-b46a-5ec139287351-kubelet-config\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:37.627600 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627339 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-systemd-units\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.627600 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-etc-openvswitch\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.627600 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627367 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-node-log\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.627600 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627380 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-ovn-node-metrics-cert\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.627600 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627401 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-systemd-units\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.627600 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627405 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0137d1fc-8fa3-4b0e-8821-4776913088f7-cni-binary-copy\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.627600 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627426 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-etc-openvswitch\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.628191 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627431 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-node-log\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.628191 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627458 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a99706d1-d012-4dd9-b46a-5ec139287351-kubelet-config\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:37.628191 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627464 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-ovnkube-config\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.628191 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627557 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-conf-dir\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.628191 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627578 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swthh\" (UniqueName: \"kubernetes.io/projected/5b91b1fa-0886-4a2c-8005-bd6f77666874-kube-api-access-swthh\") pod \"iptables-alerter-wzxhr\" (UID: \"5b91b1fa-0886-4a2c-8005-bd6f77666874\") " pod="openshift-network-operator/iptables-alerter-wzxhr" Oct 30 20:37:37.628191 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ce76c71-8b23-49b4-a61c-9b14adea86c7-cnibin\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.628191 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627607 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-cni-dir\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.628191 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627623 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-var-lib-cni-multus\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.628191 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627638 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c57ecae-89cb-4901-8f9e-e4586771f5d8-agent-certs\") pod \"konnectivity-agent-grp6n\" (UID: \"4c57ecae-89cb-4901-8f9e-e4586771f5d8\") " pod="kube-system/konnectivity-agent-grp6n" Oct 30 20:37:37.628191 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627676 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-kubernetes\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.628191 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627788 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.628191 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627803 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-sys-fs\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.628191 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627818 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-slash\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.628191 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627832 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-tuned\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.628191 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627855 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkqcq\" (UniqueName: \"kubernetes.io/projected/9390b293-dd28-4713-8452-b9fb5d7ccaec-kube-api-access-lkqcq\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.628191 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627874 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-slash\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.628191 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627868 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 20:37:37.628627 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.627901 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab625ab1-de4f-4418-bd65-edb628d963fe-tmp-dir\") pod \"node-resolver-vcndb\" (UID: \"ab625ab1-de4f-4418-bd65-edb628d963fe\") " pod="openshift-dns/node-resolver-vcndb" Oct 30 20:37:37.631402 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.631383 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-ovn-node-metrics-cert\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.631461 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.631400 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c57ecae-89cb-4901-8f9e-e4586771f5d8-agent-certs\") pod \"konnectivity-agent-grp6n\" (UID: \"4c57ecae-89cb-4901-8f9e-e4586771f5d8\") " pod="kube-system/konnectivity-agent-grp6n" Oct 30 20:37:37.642384 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.642355 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7662k\" (UniqueName: \"kubernetes.io/projected/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-kube-api-access-7662k\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.672920 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.672875 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal" podStartSLOduration=1.67286446 podStartE2EDuration="1.67286446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:37.672779792 +0000 UTC m=+16.680028095" watchObservedRunningTime="2026-10-30 20:37:37.67286446 +0000 UTC m=+16.680112823" Oct 30 20:37:37.728132 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v69k5\" (UniqueName: \"kubernetes.io/projected/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-kube-api-access-v69k5\") pod \"network-metrics-daemon-48zsl\" (UID: \"51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249\") " pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:37.728260 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mcpr\" (UniqueName: \"kubernetes.io/projected/6ce76c71-8b23-49b4-a61c-9b14adea86c7-kube-api-access-5mcpr\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.728260 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728155 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-var-lib-cni-bin\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.728260 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728178 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-var-lib-kubelet\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.728260 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728222 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-var-lib-kubelet\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.728260 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728235 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-var-lib-cni-bin\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.728411 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728345 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-daemon-config\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.728411 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728374 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a97f29d4-5d15-432a-a493-6011cf8e059e-sys\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.728411 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728398 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.728504 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-sysctl-d\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.728536 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-sysctl-conf\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.728586 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728570 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-sysctl-d\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.728621 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bsb9\" (UniqueName: \"kubernetes.io/projected/313047e7-bc02-42eb-933b-ab52575af3f7-kube-api-access-9bsb9\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.728621 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728613 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ce76c71-8b23-49b4-a61c-9b14adea86c7-system-cni-dir\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.728695 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728623 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-sysctl-conf\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.728695 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728641 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-sysconfig\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.728695 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728668 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ce76c71-8b23-49b4-a61c-9b14adea86c7-system-cni-dir\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.728856 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728669 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-registration-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.728856 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728722 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-registration-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.728856 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728720 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-sysconfig\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.728856 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728743 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-device-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.728856 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728784 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da9ceed2-3ab8-43dc-88a3-f20d7351ac30-serviceca\") pod \"node-ca-554jb\" (UID: \"da9ceed2-3ab8-43dc-88a3-f20d7351ac30\") " pod="openshift-image-registry/node-ca-554jb" Oct 30 20:37:37.728856 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728804 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-system-cni-dir\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.728856 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728807 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-device-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.728856 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728819 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-hostroot\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.728856 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728851 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-hostroot\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.729261 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728873 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-daemon-config\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.729261 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728877 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-textfile\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.729261 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728914 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-system-cni-dir\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.729261 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9390b293-dd28-4713-8452-b9fb5d7ccaec-tmp\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.729261 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728966 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.729261 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.728993 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7456\" (UniqueName: \"kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456\") pod \"network-check-target-qs2dh\" (UID: \"0832230b-2d6a-40f2-8d5f-dc1da218c7b8\") " pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:37:37.729261 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729010 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-cnibin\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.729261 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729020 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.729261 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729033 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-host\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.729261 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729055 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab625ab1-de4f-4418-bd65-edb628d963fe-hosts-file\") pod \"node-resolver-vcndb\" (UID: \"ab625ab1-de4f-4418-bd65-edb628d963fe\") " pod="openshift-dns/node-resolver-vcndb" Oct 30 20:37:37.729261 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729061 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-cnibin\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.729261 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ce76c71-8b23-49b4-a61c-9b14adea86c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.729261 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729092 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-host\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.729261 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ce76c71-8b23-49b4-a61c-9b14adea86c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.729261 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729136 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-tls\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.729261 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729140 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab625ab1-de4f-4418-bd65-edb628d963fe-hosts-file\") pod \"node-resolver-vcndb\" (UID: \"ab625ab1-de4f-4418-bd65-edb628d963fe\") " pod="openshift-dns/node-resolver-vcndb" Oct 30 20:37:37.729261 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729162 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-modprobe-d\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.729861 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729185 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-systemd\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.729861 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b91b1fa-0886-4a2c-8005-bd6f77666874-iptables-alerter-script\") pod \"iptables-alerter-wzxhr\" (UID: \"5b91b1fa-0886-4a2c-8005-bd6f77666874\") " pod="openshift-network-operator/iptables-alerter-wzxhr" Oct 30 20:37:37.729861 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729248 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-run-netns\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.729861 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729283 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-systemd\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.729861 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729286 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-etc-kubernetes\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.729861 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729310 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-modprobe-d\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.729861 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729321 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-etc-kubernetes\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.729861 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729328 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbdm6\" (UniqueName: \"kubernetes.io/projected/0137d1fc-8fa3-4b0e-8821-4776913088f7-kube-api-access-rbdm6\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.729861 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729346 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-run-netns\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.729861 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729416 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0137d1fc-8fa3-4b0e-8821-4776913088f7-cni-binary-copy\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.729861 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729442 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-conf-dir\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.729861 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swthh\" (UniqueName: \"kubernetes.io/projected/5b91b1fa-0886-4a2c-8005-bd6f77666874-kube-api-access-swthh\") pod \"iptables-alerter-wzxhr\" (UID: \"5b91b1fa-0886-4a2c-8005-bd6f77666874\") " pod="openshift-network-operator/iptables-alerter-wzxhr" Oct 30 20:37:37.729861 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729486 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ce76c71-8b23-49b4-a61c-9b14adea86c7-cnibin\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.729861 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-cni-dir\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.729861 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729514 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-var-lib-cni-multus\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.729861 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729542 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm59l\" (UniqueName: \"kubernetes.io/projected/a97f29d4-5d15-432a-a493-6011cf8e059e-kube-api-access-dm59l\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.729861 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729561 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-conf-dir\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.729861 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729586 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-kubernetes\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.730654 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729598 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-cni-dir\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.730654 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729611 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.730654 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729602 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-var-lib-cni-multus\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.730654 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729632 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ce76c71-8b23-49b4-a61c-9b14adea86c7-cnibin\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.730654 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-sys-fs\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.730654 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729674 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a97f29d4-5d15-432a-a493-6011cf8e059e-root\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.730654 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729684 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.730654 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729685 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-kubernetes\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.730654 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729688 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-sys-fs\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.730654 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.729702 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-tuned\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.730654 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730027 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da9ceed2-3ab8-43dc-88a3-f20d7351ac30-serviceca\") pod \"node-ca-554jb\" (UID: \"da9ceed2-3ab8-43dc-88a3-f20d7351ac30\") " pod="openshift-image-registry/node-ca-554jb" Oct 30 20:37:37.730654 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730233 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0137d1fc-8fa3-4b0e-8821-4776913088f7-cni-binary-copy\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.730654 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730262 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ce76c71-8b23-49b4-a61c-9b14adea86c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.730654 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730303 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ce76c71-8b23-49b4-a61c-9b14adea86c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.730654 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730328 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b91b1fa-0886-4a2c-8005-bd6f77666874-iptables-alerter-script\") pod \"iptables-alerter-wzxhr\" (UID: \"5b91b1fa-0886-4a2c-8005-bd6f77666874\") " pod="openshift-network-operator/iptables-alerter-wzxhr" Oct 30 20:37:37.730654 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730344 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkqcq\" (UniqueName: \"kubernetes.io/projected/9390b293-dd28-4713-8452-b9fb5d7ccaec-kube-api-access-lkqcq\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.730654 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab625ab1-de4f-4418-bd65-edb628d963fe-tmp-dir\") pod \"node-resolver-vcndb\" (UID: \"ab625ab1-de4f-4418-bd65-edb628d963fe\") " pod="openshift-dns/node-resolver-vcndb" Oct 30 20:37:37.731456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730422 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-accelerators-collector-config\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.731456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ce76c71-8b23-49b4-a61c-9b14adea86c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.731456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730478 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-var-lib-kubelet\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.731456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730550 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-var-lib-kubelet\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.731456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730551 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-run-k8s-cni-cncf-io\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.731456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730595 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-wtmp\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.731456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730614 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-run-k8s-cni-cncf-io\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.731456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730627 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-run\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.731456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730646 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab625ab1-de4f-4418-bd65-edb628d963fe-tmp-dir\") pod \"node-resolver-vcndb\" (UID: \"ab625ab1-de4f-4418-bd65-edb628d963fe\") " pod="openshift-dns/node-resolver-vcndb" Oct 30 20:37:37.731456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730673 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7krsp\" (UniqueName: \"kubernetes.io/projected/ab625ab1-de4f-4418-bd65-edb628d963fe-kube-api-access-7krsp\") pod \"node-resolver-vcndb\" (UID: \"ab625ab1-de4f-4418-bd65-edb628d963fe\") " pod="openshift-dns/node-resolver-vcndb" Oct 30 20:37:37.731456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730709 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da9ceed2-3ab8-43dc-88a3-f20d7351ac30-host\") pod \"node-ca-554jb\" (UID: \"da9ceed2-3ab8-43dc-88a3-f20d7351ac30\") " pod="openshift-image-registry/node-ca-554jb" Oct 30 20:37:37.731456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730742 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da9ceed2-3ab8-43dc-88a3-f20d7351ac30-host\") pod \"node-ca-554jb\" (UID: \"da9ceed2-3ab8-43dc-88a3-f20d7351ac30\") " pod="openshift-image-registry/node-ca-554jb" Oct 30 20:37:37.731456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730794 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-run\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.731456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-run-multus-certs\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.731456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730848 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-sys\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.731456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730883 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-sys\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.731456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730884 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-lib-modules\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.731456 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730852 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-run-multus-certs\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.732374 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730909 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ce76c71-8b23-49b4-a61c-9b14adea86c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.732374 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-socket-dir-parent\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.732374 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730959 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b91b1fa-0886-4a2c-8005-bd6f77666874-host-slash\") pod \"iptables-alerter-wzxhr\" (UID: \"5b91b1fa-0886-4a2c-8005-bd6f77666874\") " pod="openshift-network-operator/iptables-alerter-wzxhr" Oct 30 20:37:37.732374 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.730982 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a97f29d4-5d15-432a-a493-6011cf8e059e-metrics-client-ca\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.732374 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.731001 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-socket-dir-parent\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.732374 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.731020 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-lib-modules\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.732374 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.731045 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b91b1fa-0886-4a2c-8005-bd6f77666874-host-slash\") pod \"iptables-alerter-wzxhr\" (UID: \"5b91b1fa-0886-4a2c-8005-bd6f77666874\") " pod="openshift-network-operator/iptables-alerter-wzxhr" Oct 30 20:37:37.732374 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.731083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-socket-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.732374 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.731102 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j47v7\" (UniqueName: \"kubernetes.io/projected/da9ceed2-3ab8-43dc-88a3-f20d7351ac30-kube-api-access-j47v7\") pod \"node-ca-554jb\" (UID: \"da9ceed2-3ab8-43dc-88a3-f20d7351ac30\") " pod="openshift-image-registry/node-ca-554jb" Oct 30 20:37:37.732374 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.731150 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs\") pod \"network-metrics-daemon-48zsl\" (UID: \"51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249\") " pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:37.732374 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.731159 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ce76c71-8b23-49b4-a61c-9b14adea86c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.732374 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.731208 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-socket-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.732374 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:37.731380 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:37.732374 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.731397 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9390b293-dd28-4713-8452-b9fb5d7ccaec-tmp\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.732374 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:37.731435 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs podName:51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:38.231418068 +0000 UTC m=+17.238666371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs") pod "network-metrics-daemon-48zsl" (UID: "51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:37.732374 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.731466 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ce76c71-8b23-49b4-a61c-9b14adea86c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.732872 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.732584 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-tuned\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.736867 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:37.736848 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:37:37.736942 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:37.736875 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:37:37.736942 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:37.736895 2580 projected.go:196] Error preparing data for projected volume kube-api-access-v7456 for pod openshift-network-diagnostics/network-check-target-qs2dh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:37.737026 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:37.736957 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456 podName:0832230b-2d6a-40f2-8d5f-dc1da218c7b8 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:38.236940364 +0000 UTC m=+17.244188655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v7456" (UniqueName: "kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456") pod "network-check-target-qs2dh" (UID: "0832230b-2d6a-40f2-8d5f-dc1da218c7b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:37.737842 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.737823 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mcpr\" (UniqueName: \"kubernetes.io/projected/6ce76c71-8b23-49b4-a61c-9b14adea86c7-kube-api-access-5mcpr\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.738266 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.738250 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v69k5\" (UniqueName: \"kubernetes.io/projected/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-kube-api-access-v69k5\") pod \"network-metrics-daemon-48zsl\" (UID: \"51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249\") " pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:37.738473 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.738457 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bsb9\" (UniqueName: \"kubernetes.io/projected/313047e7-bc02-42eb-933b-ab52575af3f7-kube-api-access-9bsb9\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.741659 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.741632 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swthh\" (UniqueName: \"kubernetes.io/projected/5b91b1fa-0886-4a2c-8005-bd6f77666874-kube-api-access-swthh\") pod \"iptables-alerter-wzxhr\" (UID: \"5b91b1fa-0886-4a2c-8005-bd6f77666874\") " pod="openshift-network-operator/iptables-alerter-wzxhr" Oct 30 20:37:37.741780 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.741748 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbdm6\" (UniqueName: \"kubernetes.io/projected/0137d1fc-8fa3-4b0e-8821-4776913088f7-kube-api-access-rbdm6\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.742120 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.742105 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkqcq\" (UniqueName: \"kubernetes.io/projected/9390b293-dd28-4713-8452-b9fb5d7ccaec-kube-api-access-lkqcq\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.744387 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.744369 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7krsp\" (UniqueName: \"kubernetes.io/projected/ab625ab1-de4f-4418-bd65-edb628d963fe-kube-api-access-7krsp\") pod \"node-resolver-vcndb\" (UID: \"ab625ab1-de4f-4418-bd65-edb628d963fe\") " pod="openshift-dns/node-resolver-vcndb" Oct 30 20:37:37.745664 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.745646 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j47v7\" (UniqueName: \"kubernetes.io/projected/da9ceed2-3ab8-43dc-88a3-f20d7351ac30-kube-api-access-j47v7\") pod \"node-ca-554jb\" (UID: \"da9ceed2-3ab8-43dc-88a3-f20d7351ac30\") " pod="openshift-image-registry/node-ca-554jb" Oct 30 20:37:37.751576 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.751559 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-grp6n" Oct 30 20:37:37.761693 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:37.761670 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c57ecae_89cb_4901_8f9e_e4586771f5d8.slice/crio-ebad2169c8c3c0fbac5433d4c72760bf94ff4a03186a0c4be91d54b618706cef WatchSource:0}: Error finding container ebad2169c8c3c0fbac5433d4c72760bf94ff4a03186a0c4be91d54b618706cef: Status 404 returned error can't find the container with id ebad2169c8c3c0fbac5433d4c72760bf94ff4a03186a0c4be91d54b618706cef Oct 30 20:37:37.769470 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.769456 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vcndb" Oct 30 20:37:37.772092 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.772077 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2kvw6" Oct 30 20:37:37.778093 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.777995 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:37.780395 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:37.780372 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab625ab1_de4f_4418_bd65_edb628d963fe.slice/crio-31b3722fcbc274c096b600beb5b1892800474f1df99d66e110a6f9b8929fd4a0 WatchSource:0}: Error finding container 31b3722fcbc274c096b600beb5b1892800474f1df99d66e110a6f9b8929fd4a0: Status 404 returned error can't find the container with id 31b3722fcbc274c096b600beb5b1892800474f1df99d66e110a6f9b8929fd4a0 Oct 30 20:37:37.780920 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:37.780896 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0137d1fc_8fa3_4b0e_8821_4776913088f7.slice/crio-e7d808b3d42ff570260ca243f48eef40b51f28d66b796b3b743f4fb5b2498ad1 WatchSource:0}: Error finding container e7d808b3d42ff570260ca243f48eef40b51f28d66b796b3b743f4fb5b2498ad1: Status 404 returned error can't find the container with id e7d808b3d42ff570260ca243f48eef40b51f28d66b796b3b743f4fb5b2498ad1 Oct 30 20:37:37.784940 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.784918 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:37:37.786169 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:37.786151 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9224f6c0_bc3e_4eee_a78b_431e6c8f4705.slice/crio-4de8b4f4919804deda81fcd89fb21765d52641e297c0176b000d39775f8d57f7 WatchSource:0}: Error finding container 4de8b4f4919804deda81fcd89fb21765d52641e297c0176b000d39775f8d57f7: Status 404 returned error can't find the container with id 4de8b4f4919804deda81fcd89fb21765d52641e297c0176b000d39775f8d57f7 Oct 30 20:37:37.786617 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.786598 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-554jb" Oct 30 20:37:37.789501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.789485 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:37:37.792624 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.792604 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wzxhr" Oct 30 20:37:37.795951 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:37.795928 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313047e7_bc02_42eb_933b_ab52575af3f7.slice/crio-89858e985189f0e775c5a1b655747c867d52234b23c1f97cf54f3636fbc9e0f7 WatchSource:0}: Error finding container 89858e985189f0e775c5a1b655747c867d52234b23c1f97cf54f3636fbc9e0f7: Status 404 returned error can't find the container with id 89858e985189f0e775c5a1b655747c867d52234b23c1f97cf54f3636fbc9e0f7 Oct 30 20:37:37.797583 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:37.797567 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9ceed2_3ab8_43dc_88a3_f20d7351ac30.slice/crio-c5d7feb6f3ddb313ad1539d55cf36b8dc12079ff539dc6538e3bdb87a0144d2b WatchSource:0}: Error finding container c5d7feb6f3ddb313ad1539d55cf36b8dc12079ff539dc6538e3bdb87a0144d2b: Status 404 returned error can't find the container with id c5d7feb6f3ddb313ad1539d55cf36b8dc12079ff539dc6538e3bdb87a0144d2b Oct 30 20:37:37.802200 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:37.802181 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9390b293_dd28_4713_8452_b9fb5d7ccaec.slice/crio-345345c12e77c9aaf2dbfe638c636738b84a6393440f03bdc249c191ab67ac9d WatchSource:0}: Error finding container 345345c12e77c9aaf2dbfe638c636738b84a6393440f03bdc249c191ab67ac9d: Status 404 returned error can't find the container with id 345345c12e77c9aaf2dbfe638c636738b84a6393440f03bdc249c191ab67ac9d Oct 30 20:37:37.805069 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:37.805051 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b91b1fa_0886_4a2c_8005_bd6f77666874.slice/crio-66adbecdad87f0fa75c8faa791afe3b11c48828e61636773b808a946920c932e WatchSource:0}: Error finding container 66adbecdad87f0fa75c8faa791afe3b11c48828e61636773b808a946920c932e: Status 404 returned error can't find the container with id 66adbecdad87f0fa75c8faa791afe3b11c48828e61636773b808a946920c932e Oct 30 20:37:37.808589 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.808572 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:37:37.817682 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:37.817662 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce76c71_8b23_49b4_a61c_9b14adea86c7.slice/crio-755525b3aeedd7c5f9ac6757f8ed83741be4815d1cb49602e62cc30d9344f271 WatchSource:0}: Error finding container 755525b3aeedd7c5f9ac6757f8ed83741be4815d1cb49602e62cc30d9344f271: Status 404 returned error can't find the container with id 755525b3aeedd7c5f9ac6757f8ed83741be4815d1cb49602e62cc30d9344f271 Oct 30 20:37:37.831645 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.831588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-wtmp\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.831645 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.831621 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a97f29d4-5d15-432a-a493-6011cf8e059e-metrics-client-ca\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.831795 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.831648 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a97f29d4-5d15-432a-a493-6011cf8e059e-sys\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.831795 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.831669 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.831795 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.831697 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-textfile\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.831795 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.831719 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a97f29d4-5d15-432a-a493-6011cf8e059e-sys\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.831795 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.831741 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-wtmp\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.831795 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.831724 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-tls\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.832074 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.831854 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm59l\" (UniqueName: \"kubernetes.io/projected/a97f29d4-5d15-432a-a493-6011cf8e059e-kube-api-access-dm59l\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.832074 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.831889 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a97f29d4-5d15-432a-a493-6011cf8e059e-root\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.832074 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.831918 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-accelerators-collector-config\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.832074 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.831951 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a97f29d4-5d15-432a-a493-6011cf8e059e-root\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.832745 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.832723 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-textfile\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.832915 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.832896 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a97f29d4-5d15-432a-a493-6011cf8e059e-metrics-client-ca\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.833061 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.833043 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-accelerators-collector-config\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.834399 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.834382 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-tls\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.835589 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.835572 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.840449 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.840431 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm59l\" (UniqueName: \"kubernetes.io/projected/a97f29d4-5d15-432a-a493-6011cf8e059e-kube-api-access-dm59l\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.879437 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:37.879411 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:37:37.888827 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:37:37.888804 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97f29d4_5d15_432a_a493_6011cf8e059e.slice/crio-0eeca3a5e349b85acd986b5c4feba101508cbd8b9205aa426f4b34ae31975955 WatchSource:0}: Error finding container 0eeca3a5e349b85acd986b5c4feba101508cbd8b9205aa426f4b34ae31975955: Status 404 returned error can't find the container with id 0eeca3a5e349b85acd986b5c4feba101508cbd8b9205aa426f4b34ae31975955 Oct 30 20:37:38.134336 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:38.134265 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:38.134477 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:38.134359 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:38.134477 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:38.134416 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret podName:a99706d1-d012-4dd9-b46a-5ec139287351 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:39.134403492 +0000 UTC m=+18.141651781 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret") pod "global-pull-secret-syncer-5wp5n" (UID: "a99706d1-d012-4dd9-b46a-5ec139287351") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:38.235042 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:38.235014 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs\") pod \"network-metrics-daemon-48zsl\" (UID: \"51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249\") " pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:38.235220 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:38.235161 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:38.235284 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:38.235228 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs podName:51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:39.235209277 +0000 UTC m=+18.242457587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs") pod "network-metrics-daemon-48zsl" (UID: "51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:38.336394 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:38.336350 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7456\" (UniqueName: \"kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456\") pod \"network-check-target-qs2dh\" (UID: \"0832230b-2d6a-40f2-8d5f-dc1da218c7b8\") " pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:37:38.336939 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:38.336514 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:37:38.336939 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:38.336531 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:37:38.336939 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:38.336544 2580 projected.go:196] Error preparing data for projected volume kube-api-access-v7456 for pod openshift-network-diagnostics/network-check-target-qs2dh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:38.336939 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:38.336598 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456 podName:0832230b-2d6a-40f2-8d5f-dc1da218c7b8 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:39.336581533 +0000 UTC m=+18.343829837 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v7456" (UniqueName: "kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456") pod "network-check-target-qs2dh" (UID: "0832230b-2d6a-40f2-8d5f-dc1da218c7b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:38.436189 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:38.436080 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:32:37 +0000 UTC" deadline="2028-07-16 20:53:07.804714492 +0000 UTC" Oct 30 20:37:38.436189 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:38.436144 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15000h15m29.368574771s" Oct 30 20:37:38.520173 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:38.519848 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vcndb" event={"ID":"ab625ab1-de4f-4418-bd65-edb628d963fe","Type":"ContainerStarted","Data":"31b3722fcbc274c096b600beb5b1892800474f1df99d66e110a6f9b8929fd4a0"} Oct 30 20:37:38.542515 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:38.542423 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-grp6n" event={"ID":"4c57ecae-89cb-4901-8f9e-e4586771f5d8","Type":"ContainerStarted","Data":"ebad2169c8c3c0fbac5433d4c72760bf94ff4a03186a0c4be91d54b618706cef"} Oct 30 20:37:38.562235 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:38.562141 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n7lg7" event={"ID":"6ce76c71-8b23-49b4-a61c-9b14adea86c7","Type":"ContainerStarted","Data":"755525b3aeedd7c5f9ac6757f8ed83741be4815d1cb49602e62cc30d9344f271"} Oct 30 20:37:38.572517 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:38.572451 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wr47v" event={"ID":"9390b293-dd28-4713-8452-b9fb5d7ccaec","Type":"ContainerStarted","Data":"345345c12e77c9aaf2dbfe638c636738b84a6393440f03bdc249c191ab67ac9d"} Oct 30 20:37:38.594382 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:38.594326 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-554jb" event={"ID":"da9ceed2-3ab8-43dc-88a3-f20d7351ac30","Type":"ContainerStarted","Data":"c5d7feb6f3ddb313ad1539d55cf36b8dc12079ff539dc6538e3bdb87a0144d2b"} Oct 30 20:37:38.625884 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:38.625852 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" event={"ID":"9224f6c0-bc3e-4eee-a78b-431e6c8f4705","Type":"ContainerStarted","Data":"4de8b4f4919804deda81fcd89fb21765d52641e297c0176b000d39775f8d57f7"} Oct 30 20:37:38.639258 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:38.639202 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2kvw6" event={"ID":"0137d1fc-8fa3-4b0e-8821-4776913088f7","Type":"ContainerStarted","Data":"e7d808b3d42ff570260ca243f48eef40b51f28d66b796b3b743f4fb5b2498ad1"} Oct 30 20:37:38.652860 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:38.652837 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal_0459ad6c6db5719be39646515856f6a3/setup/0.log" Oct 30 20:37:38.654605 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:38.654583 2580 generic.go:356] "Generic (PLEG): container finished" podID="0459ad6c6db5719be39646515856f6a3" containerID="86c55d4aea6a923581be42279421c6a7d7627e21c4599cc522d38f8b2bee8cda" exitCode=0 Oct 30 20:37:38.654824 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:38.654728 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" event={"ID":"0459ad6c6db5719be39646515856f6a3","Type":"ContainerDied","Data":"86c55d4aea6a923581be42279421c6a7d7627e21c4599cc522d38f8b2bee8cda"} Oct 30 20:37:38.654983 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:38.654973 2580 scope.go:122] "RemoveContainer" containerID="e73fb87d6e8625c463b8e413b77079eebbf77689c5517adebe0f692312539cc5" Oct 30 20:37:38.655881 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:38.655866 2580 scope.go:122] "RemoveContainer" containerID="e73fb87d6e8625c463b8e413b77079eebbf77689c5517adebe0f692312539cc5" Oct 30 20:37:38.669334 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:38.669247 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bqhgw" event={"ID":"a97f29d4-5d15-432a-a493-6011cf8e059e","Type":"ContainerStarted","Data":"0eeca3a5e349b85acd986b5c4feba101508cbd8b9205aa426f4b34ae31975955"} Oct 30 20:37:38.685565 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:38.683909 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wzxhr" event={"ID":"5b91b1fa-0886-4a2c-8005-bd6f77666874","Type":"ContainerStarted","Data":"66adbecdad87f0fa75c8faa791afe3b11c48828e61636773b808a946920c932e"} Oct 30 20:37:38.697842 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:38.697809 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" event={"ID":"313047e7-bc02-42eb-933b-ab52575af3f7","Type":"ContainerStarted","Data":"89858e985189f0e775c5a1b655747c867d52234b23c1f97cf54f3636fbc9e0f7"} Oct 30 20:37:38.713004 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:38.712965 2580 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal_openshift-machine-config-operator_0459ad6c6db5719be39646515856f6a3_0 in pod sandbox 6699dc1c4ebf7b70abd16bb5b79b34127d1801b261cf3b6e50ad3abbaee3b1db from index: no such id: 'e73fb87d6e8625c463b8e413b77079eebbf77689c5517adebe0f692312539cc5'" containerID="e73fb87d6e8625c463b8e413b77079eebbf77689c5517adebe0f692312539cc5" Oct 30 20:37:38.713128 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:38.713047 2580 kuberuntime_container.go:987] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal_openshift-machine-config-operator_0459ad6c6db5719be39646515856f6a3_0 in pod sandbox 6699dc1c4ebf7b70abd16bb5b79b34127d1801b261cf3b6e50ad3abbaee3b1db from index: no such id: 'e73fb87d6e8625c463b8e413b77079eebbf77689c5517adebe0f692312539cc5'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal_openshift-machine-config-operator(0459ad6c6db5719be39646515856f6a3)\"" logger="UnhandledError" Oct 30 20:37:39.142169 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:39.142127 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:39.142350 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:39.142303 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:39.142413 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:39.142363 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret podName:a99706d1-d012-4dd9-b46a-5ec139287351 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:41.142346163 +0000 UTC m=+20.149594466 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret") pod "global-pull-secret-syncer-5wp5n" (UID: "a99706d1-d012-4dd9-b46a-5ec139287351") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:39.243582 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:39.243500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs\") pod \"network-metrics-daemon-48zsl\" (UID: \"51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249\") " pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:39.243734 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:39.243678 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:39.243813 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:39.243753 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs podName:51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:41.243732159 +0000 UTC m=+20.250980470 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs") pod "network-metrics-daemon-48zsl" (UID: "51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:39.345040 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:39.345003 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7456\" (UniqueName: \"kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456\") pod \"network-check-target-qs2dh\" (UID: \"0832230b-2d6a-40f2-8d5f-dc1da218c7b8\") " pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:37:39.345442 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:39.345199 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:37:39.345442 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:39.345222 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:37:39.345442 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:39.345235 2580 projected.go:196] Error preparing data for projected volume kube-api-access-v7456 for pod openshift-network-diagnostics/network-check-target-qs2dh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:39.345442 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:39.345291 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456 podName:0832230b-2d6a-40f2-8d5f-dc1da218c7b8 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:41.345272385 +0000 UTC m=+20.352520709 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v7456" (UniqueName: "kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456") pod "network-check-target-qs2dh" (UID: "0832230b-2d6a-40f2-8d5f-dc1da218c7b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:39.436355 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:39.436288 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:32:37 +0000 UTC" deadline="2028-05-17 18:18:45.838851957 +0000 UTC" Oct 30 20:37:39.436355 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:39.436354 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13557h41m6.402502454s" Oct 30 20:37:39.476841 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:39.476810 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:39.477007 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:39.476952 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48zsl" podUID="51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249" Oct 30 20:37:39.477356 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:39.477333 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:39.477451 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:39.477432 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wp5n" podUID="a99706d1-d012-4dd9-b46a-5ec139287351" Oct 30 20:37:39.477511 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:39.477466 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:37:39.477559 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:39.477531 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs2dh" podUID="0832230b-2d6a-40f2-8d5f-dc1da218c7b8" Oct 30 20:37:39.712162 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:39.712075 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" event={"ID":"0459ad6c6db5719be39646515856f6a3","Type":"ContainerStarted","Data":"885f87af00e1312a5de4e66c046c8dcd02f79954e5a2083be4e8cc7c1aa77a67"} Oct 30 20:37:39.817213 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:39.817010 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:37:40.720879 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:40.720008 2580 generic.go:356] "Generic (PLEG): container finished" podID="a97f29d4-5d15-432a-a493-6011cf8e059e" containerID="f90eb0ea414935aebd2e8458d6fce14a09e5f821d6732cf3d219dfd3b1bfb2f6" exitCode=0 Oct 30 20:37:40.720879 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:40.720073 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bqhgw" event={"ID":"a97f29d4-5d15-432a-a493-6011cf8e059e","Type":"ContainerDied","Data":"f90eb0ea414935aebd2e8458d6fce14a09e5f821d6732cf3d219dfd3b1bfb2f6"} Oct 30 20:37:40.740034 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:40.739458 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" podStartSLOduration=4.7394401219999995 podStartE2EDuration="4.739440122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:39.735870838 +0000 UTC m=+18.743119150" watchObservedRunningTime="2026-10-30 20:37:40.739440122 +0000 UTC m=+19.746688436" Oct 30 20:37:41.161835 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:41.161575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:41.162007 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:41.161961 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:41.162077 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:41.162026 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret podName:a99706d1-d012-4dd9-b46a-5ec139287351 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:45.162008583 +0000 UTC m=+24.169256886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret") pod "global-pull-secret-syncer-5wp5n" (UID: "a99706d1-d012-4dd9-b46a-5ec139287351") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:41.263164 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:41.263084 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs\") pod \"network-metrics-daemon-48zsl\" (UID: \"51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249\") " pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:41.263313 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:41.263242 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:41.263313 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:41.263305 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs podName:51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:45.26328669 +0000 UTC m=+24.270534984 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs") pod "network-metrics-daemon-48zsl" (UID: "51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:41.363813 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:41.363748 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7456\" (UniqueName: \"kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456\") pod \"network-check-target-qs2dh\" (UID: \"0832230b-2d6a-40f2-8d5f-dc1da218c7b8\") " pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:37:41.364144 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:41.364096 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:37:41.364144 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:41.364140 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:37:41.364298 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:41.364153 2580 projected.go:196] Error preparing data for projected volume kube-api-access-v7456 for pod openshift-network-diagnostics/network-check-target-qs2dh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:41.364298 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:41.364249 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456 podName:0832230b-2d6a-40f2-8d5f-dc1da218c7b8 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:45.364223755 +0000 UTC m=+24.371472056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v7456" (UniqueName: "kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456") pod "network-check-target-qs2dh" (UID: "0832230b-2d6a-40f2-8d5f-dc1da218c7b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:41.482585 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:41.482548 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:37:41.482761 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:41.482548 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:41.482761 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:41.482679 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs2dh" podUID="0832230b-2d6a-40f2-8d5f-dc1da218c7b8" Oct 30 20:37:41.482761 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:41.482705 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:41.482944 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:41.482819 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wp5n" podUID="a99706d1-d012-4dd9-b46a-5ec139287351" Oct 30 20:37:41.482944 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:41.482908 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48zsl" podUID="51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249" Oct 30 20:37:41.728370 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:41.728335 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bqhgw" event={"ID":"a97f29d4-5d15-432a-a493-6011cf8e059e","Type":"ContainerStarted","Data":"c53db7ac4743796ca92aaa242e98238391a098c0cecaeb9b2bace73bcf14bcfb"} Oct 30 20:37:41.728749 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:41.728400 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bqhgw" event={"ID":"a97f29d4-5d15-432a-a493-6011cf8e059e","Type":"ContainerStarted","Data":"2cc63a6efa632edaca1e8fe4a11d04784120f343b04122de978b1e898a24e5e6"} Oct 30 20:37:41.736086 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:41.736061 2580 generic.go:356] "Generic (PLEG): container finished" podID="6ce76c71-8b23-49b4-a61c-9b14adea86c7" containerID="da08cdc8b0c8d5eeddea290c41ae4d31fb25ab9fb6dd3353abfdd7af00dfd5b9" exitCode=0 Oct 30 20:37:41.736138 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:41.736108 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n7lg7" event={"ID":"6ce76c71-8b23-49b4-a61c-9b14adea86c7","Type":"ContainerDied","Data":"da08cdc8b0c8d5eeddea290c41ae4d31fb25ab9fb6dd3353abfdd7af00dfd5b9"} Oct 30 20:37:41.766130 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:41.766073 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bqhgw" podStartSLOduration=2.142581659 podStartE2EDuration="4.766053103s" totalImagesPullingTime="2.53308534s" totalInitContainerRuntime="90.386104ms" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:41.746540543 +0000 UTC m=+20.753788848" watchObservedRunningTime="2026-10-30 20:37:41.766053103 +0000 UTC m=+20.773301414" Oct 30 20:37:42.748913 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:42.748881 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-grp6n" event={"ID":"4c57ecae-89cb-4901-8f9e-e4586771f5d8","Type":"ContainerStarted","Data":"c7e24adc5864125797c6db2ee70eeb31c75d3b423ba87b003768922817fece82"} Oct 30 20:37:42.752529 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:42.752508 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-grp6n" Oct 30 20:37:42.752953 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:42.752936 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-grp6n" Oct 30 20:37:42.788388 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:42.788329 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-grp6n" podStartSLOduration=4.822655549 podStartE2EDuration="8.788310057s" totalImagesPullingTime="3.965654508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:42.76979642 +0000 UTC m=+21.777044731" watchObservedRunningTime="2026-10-30 20:37:42.788310057 +0000 UTC m=+21.795558370" Oct 30 20:37:43.477530 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:43.476707 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:37:43.477530 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:43.476852 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs2dh" podUID="0832230b-2d6a-40f2-8d5f-dc1da218c7b8" Oct 30 20:37:43.477530 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:43.477210 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:43.477530 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:43.477315 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48zsl" podUID="51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249" Oct 30 20:37:43.477530 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:43.477361 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:43.477530 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:43.477426 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wp5n" podUID="a99706d1-d012-4dd9-b46a-5ec139287351" Oct 30 20:37:43.753728 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:43.753646 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-554jb" event={"ID":"da9ceed2-3ab8-43dc-88a3-f20d7351ac30","Type":"ContainerStarted","Data":"82ff6a257bcf56eef2bc7c44eb12d4c20a5e74fcaf9bd3fd96ec1ebeef11f14f"} Oct 30 20:37:43.796001 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:43.795959 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grp6n_4c57ecae-89cb-4901-8f9e-e4586771f5d8/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:43.882891 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:43.882811 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grp6n_4c57ecae-89cb-4901-8f9e-e4586771f5d8/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:44.021485 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:44.021402 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grp6n_4c57ecae-89cb-4901-8f9e-e4586771f5d8/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:44.130484 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:44.130451 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grp6n_4c57ecae-89cb-4901-8f9e-e4586771f5d8/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:44.759717 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:44.759205 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" event={"ID":"313047e7-bc02-42eb-933b-ab52575af3f7","Type":"ContainerStarted","Data":"19a0186286209eb97441c726439473e2e465c4191e22ac7f23c6285fa5f16c0a"} Oct 30 20:37:44.759717 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:44.759343 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:37:45.198642 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:45.197797 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:45.198642 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:45.197968 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:45.198642 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:45.198022 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret podName:a99706d1-d012-4dd9-b46a-5ec139287351 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:53.198007324 +0000 UTC m=+32.205255627 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret") pod "global-pull-secret-syncer-5wp5n" (UID: "a99706d1-d012-4dd9-b46a-5ec139287351") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:45.299064 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:45.299031 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs\") pod \"network-metrics-daemon-48zsl\" (UID: \"51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249\") " pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:45.299181 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:45.299142 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:45.299237 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:45.299211 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs podName:51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:53.299190163 +0000 UTC m=+32.306438466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs") pod "network-metrics-daemon-48zsl" (UID: "51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:45.400690 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:45.400029 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7456\" (UniqueName: \"kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456\") pod \"network-check-target-qs2dh\" (UID: \"0832230b-2d6a-40f2-8d5f-dc1da218c7b8\") " pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:37:45.400690 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:45.400194 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:37:45.400690 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:45.400217 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:37:45.400690 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:45.400230 2580 projected.go:196] Error preparing data for projected volume kube-api-access-v7456 for pod openshift-network-diagnostics/network-check-target-qs2dh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:45.400690 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:45.400299 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456 podName:0832230b-2d6a-40f2-8d5f-dc1da218c7b8 nodeName:}" failed. No retries permitted until 2026-10-30 20:37:53.400272955 +0000 UTC m=+32.407521269 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v7456" (UniqueName: "kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456") pod "network-check-target-qs2dh" (UID: "0832230b-2d6a-40f2-8d5f-dc1da218c7b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:45.477801 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:45.477685 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:45.477933 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:45.477805 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wp5n" podUID="a99706d1-d012-4dd9-b46a-5ec139287351" Oct 30 20:37:45.477933 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:45.477875 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:37:45.478054 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:45.477945 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs2dh" podUID="0832230b-2d6a-40f2-8d5f-dc1da218c7b8" Oct 30 20:37:45.478054 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:45.478006 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:45.478151 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:45.478087 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48zsl" podUID="51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249" Oct 30 20:37:45.512136 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:45.512112 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 20:37:45.765234 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:45.765147 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2kvw6" event={"ID":"0137d1fc-8fa3-4b0e-8821-4776913088f7","Type":"ContainerStarted","Data":"61e7e333098c8a0c73f177208c281c95dd835e3109022ebdee3658d9eba40859"} Oct 30 20:37:45.769759 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:45.769729 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" event={"ID":"313047e7-bc02-42eb-933b-ab52575af3f7","Type":"ContainerStarted","Data":"28a7be43fdd06975ec9032da3278f730afa1ea299f5b5d56ae2562daf3a98094"} Oct 30 20:37:45.783688 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:45.783628 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-554jb" podStartSLOduration=6.701111085 podStartE2EDuration="11.783611767s" totalImagesPullingTime="5.082500682s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:43.772213701 +0000 UTC m=+22.779462013" watchObservedRunningTime="2026-10-30 20:37:45.783611767 +0000 UTC m=+24.790860077" Oct 30 20:37:45.912592 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:45.912541 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grp6n_4c57ecae-89cb-4901-8f9e-e4586771f5d8/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:46.022680 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:46.022448 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grp6n_4c57ecae-89cb-4901-8f9e-e4586771f5d8/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:46.419499 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:46.419285 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:37:45.5121333Z","UUID":"c5d510b7-55c8-4368-923f-b2d90a73ce73","Handler":null,"Name":"","Endpoint":""} Oct 30 20:37:46.421235 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:46.421197 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 20:37:46.421235 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:46.421232 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 20:37:46.712893 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:46.712821 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grp6n_4c57ecae-89cb-4901-8f9e-e4586771f5d8/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:46.775698 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:46.775661 2580 generic.go:356] "Generic (PLEG): container finished" podID="6ce76c71-8b23-49b4-a61c-9b14adea86c7" containerID="e462dd2bc35cb1bbbe0a3a8f39872f60eefa95e5d915139ff2b7ac16d78a8a0e" exitCode=0 Oct 30 20:37:46.776173 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:46.775760 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n7lg7" event={"ID":"6ce76c71-8b23-49b4-a61c-9b14adea86c7","Type":"ContainerDied","Data":"e462dd2bc35cb1bbbe0a3a8f39872f60eefa95e5d915139ff2b7ac16d78a8a0e"} Oct 30 20:37:46.779434 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:46.779302 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vcndb" event={"ID":"ab625ab1-de4f-4418-bd65-edb628d963fe","Type":"ContainerStarted","Data":"a96f5b044ab862c01be1d2c8e6cbb790e5093c6d508567c62ea84f2917b26127"} Oct 30 20:37:46.809318 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:46.809274 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-2kvw6" podStartSLOduration=5.385669412 podStartE2EDuration="12.809261564s" totalImagesPullingTime="7.423592152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:45.78395325 +0000 UTC m=+24.791201561" watchObservedRunningTime="2026-10-30 20:37:46.809261564 +0000 UTC m=+25.816509873" Oct 30 20:37:46.824697 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:46.824672 2580 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" failed. No retries permitted until 2026-10-30 20:37:47.324653647 +0000 UTC m=+26.331901951 (durationBeforeRetry 500ms). Error: RegisterPlugin error -- plugin registration failed with err: error updating CSINode object with CSI driver node info: error updating CSINode: timed out waiting for the condition; caused by: CSINode.storage.k8s.io "ip-10-0-130-52.us-east-2.compute.internal" is invalid: metadata.ownerReferences.uid: Invalid value: "": must not be empty: rpc error: code = Unavailable desc = error reading from server: EOF Oct 30 20:37:46.827147 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:46.827103 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-vcndb" podStartSLOduration=4.753634946 podStartE2EDuration="12.82708723s" totalImagesPullingTime="8.073452284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:46.826353478 +0000 UTC m=+25.833601788" watchObservedRunningTime="2026-10-30 20:37:46.82708723 +0000 UTC m=+25.834335541" Oct 30 20:37:46.832309 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:46.832233 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grp6n_4c57ecae-89cb-4901-8f9e-e4586771f5d8/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:47.478263 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:47.478183 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:47.478402 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:47.478188 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:47.478402 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:47.478196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:37:47.478402 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:47.478344 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wp5n" podUID="a99706d1-d012-4dd9-b46a-5ec139287351" Oct 30 20:37:47.478552 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:47.478400 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs2dh" podUID="0832230b-2d6a-40f2-8d5f-dc1da218c7b8" Oct 30 20:37:47.478552 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:47.478519 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48zsl" podUID="51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249" Oct 30 20:37:47.783813 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:47.783724 2580 generic.go:356] "Generic (PLEG): container finished" podID="6ce76c71-8b23-49b4-a61c-9b14adea86c7" containerID="1bc2a30768271765c99d111719a0ef275ccab7276fadd54368598bf12845129a" exitCode=0 Oct 30 20:37:47.784233 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:47.783810 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n7lg7" event={"ID":"6ce76c71-8b23-49b4-a61c-9b14adea86c7","Type":"ContainerDied","Data":"1bc2a30768271765c99d111719a0ef275ccab7276fadd54368598bf12845129a"} Oct 30 20:37:47.786518 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:47.786485 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wr47v" event={"ID":"9390b293-dd28-4713-8452-b9fb5d7ccaec","Type":"ContainerStarted","Data":"91aebc3fbd21aa7c413e6d7b4d5ccc95f9c6774b297451ba259ffb3de7b23eaf"} Oct 30 20:37:47.788028 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:47.788006 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wzxhr" event={"ID":"5b91b1fa-0886-4a2c-8005-bd6f77666874","Type":"ContainerStarted","Data":"32ee3fdfe084b2f3fef802634e2ca393fb659503ba593834f1da3d0ead7d9add"} Oct 30 20:37:47.790251 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:47.790236 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 20:37:47.790685 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:47.790647 2580 generic.go:356] "Generic (PLEG): container finished" podID="313047e7-bc02-42eb-933b-ab52575af3f7" containerID="28a7be43fdd06975ec9032da3278f730afa1ea299f5b5d56ae2562daf3a98094" exitCode=1 Oct 30 20:37:47.790802 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:47.790678 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" event={"ID":"313047e7-bc02-42eb-933b-ab52575af3f7","Type":"ContainerDied","Data":"28a7be43fdd06975ec9032da3278f730afa1ea299f5b5d56ae2562daf3a98094"} Oct 30 20:37:47.790802 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:47.790744 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" event={"ID":"313047e7-bc02-42eb-933b-ab52575af3f7","Type":"ContainerStarted","Data":"f0e746f103f1583cac6626799e456dc17acc1921c9549a0f3cbf4f6fe7e4fedc"} Oct 30 20:37:47.791000 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:47.790975 2580 scope.go:122] "RemoveContainer" containerID="28a7be43fdd06975ec9032da3278f730afa1ea299f5b5d56ae2562daf3a98094" Oct 30 20:37:47.847346 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:47.847302 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wr47v" podStartSLOduration=4.024643035 podStartE2EDuration="13.847290418s" totalImagesPullingTime="9.822647383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:47.846778646 +0000 UTC m=+26.854026955" watchObservedRunningTime="2026-10-30 20:37:47.847290418 +0000 UTC m=+26.854538727" Oct 30 20:37:47.867676 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:47.867638 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wzxhr" podStartSLOduration=5.901320121 podStartE2EDuration="13.867624637s" totalImagesPullingTime="7.966304516s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:47.867144289 +0000 UTC m=+26.874392599" watchObservedRunningTime="2026-10-30 20:37:47.867624637 +0000 UTC m=+26.874872947" Oct 30 20:37:47.895296 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:47.895274 2580 plugin_watcher.go:214] "Removing socket path from desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 20:37:47.895591 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:47.895575 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 20:37:48.009956 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:48.009914 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grp6n_4c57ecae-89cb-4901-8f9e-e4586771f5d8/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:48.108698 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:48.108625 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grp6n_4c57ecae-89cb-4901-8f9e-e4586771f5d8/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:48.420631 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:48.420480 2580 reconciler.go:150] "OperationExecutor.UnregisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"0001-01-01T00:00:00Z","UUID":"c5d510b7-55c8-4368-923f-b2d90a73ce73","Handler":{},"Name":"ebs.csi.aws.com","Endpoint":"/var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock"} Oct 30 20:37:48.420631 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:48.420558 2580 csi_plugin.go:271] kubernetes.io/csi: registrationHandler.DeRegisterPlugin request for plugin ebs.csi.aws.com, endpoint /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 20:37:48.758978 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:48.758872 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grp6n_4c57ecae-89cb-4901-8f9e-e4586771f5d8/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:48.795694 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:48.795657 2580 generic.go:356] "Generic (PLEG): container finished" podID="6ce76c71-8b23-49b4-a61c-9b14adea86c7" containerID="563d32af2643c0ad4dd76de6c65f1fc64ca8e36155d3c99127cb3b944b63fcf6" exitCode=0 Oct 30 20:37:48.796161 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:48.795745 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n7lg7" event={"ID":"6ce76c71-8b23-49b4-a61c-9b14adea86c7","Type":"ContainerDied","Data":"563d32af2643c0ad4dd76de6c65f1fc64ca8e36155d3c99127cb3b944b63fcf6"} Oct 30 20:37:48.798587 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:48.798569 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 20:37:48.798990 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:48.798961 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" event={"ID":"313047e7-bc02-42eb-933b-ab52575af3f7","Type":"ContainerStarted","Data":"7789160d176237525da0e12c3b3eaa55170e97ed76214d39f73a957e136e49be"} Oct 30 20:37:48.842581 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:48.842533 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" podStartSLOduration=6.052092184 podStartE2EDuration="14.842517996s" totalImagesPullingTime="8.790425812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:48.842117983 +0000 UTC m=+27.849366354" watchObservedRunningTime="2026-10-30 20:37:48.842517996 +0000 UTC m=+27.849766305" Oct 30 20:37:48.849610 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:48.849582 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grp6n_4c57ecae-89cb-4901-8f9e-e4586771f5d8/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:49.421565 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:49.421525 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:37:47.895589713Z","UUID":"c519e3e3-03af-4916-a823-dc4e97d779e6","Handler":null,"Name":"","Endpoint":""} Oct 30 20:37:49.423040 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:49.422807 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 20:37:49.423040 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:49.422854 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 20:37:49.474355 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:49.474323 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:49.474520 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:49.474472 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48zsl" podUID="51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249" Oct 30 20:37:49.474849 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:49.474828 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:49.474954 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:49.474931 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wp5n" podUID="a99706d1-d012-4dd9-b46a-5ec139287351" Oct 30 20:37:49.475101 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:49.475080 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:37:49.475207 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:49.475187 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs2dh" podUID="0832230b-2d6a-40f2-8d5f-dc1da218c7b8" Oct 30 20:37:50.806609 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:50.806531 2580 generic.go:356] "Generic (PLEG): container finished" podID="6ce76c71-8b23-49b4-a61c-9b14adea86c7" containerID="966df842b85647b2e8a8ff0d0ecafe62c24585be0de4c1d0712f02641b51c2b0" exitCode=0 Oct 30 20:37:50.806609 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:50.806575 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n7lg7" event={"ID":"6ce76c71-8b23-49b4-a61c-9b14adea86c7","Type":"ContainerDied","Data":"966df842b85647b2e8a8ff0d0ecafe62c24585be0de4c1d0712f02641b51c2b0"} Oct 30 20:37:51.476558 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:51.476520 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:51.476735 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:51.476676 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wp5n" podUID="a99706d1-d012-4dd9-b46a-5ec139287351" Oct 30 20:37:51.477141 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:51.477119 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:51.477366 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:51.477339 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48zsl" podUID="51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249" Oct 30 20:37:51.479386 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:51.479356 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:37:51.479934 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:51.479902 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs2dh" podUID="0832230b-2d6a-40f2-8d5f-dc1da218c7b8" Oct 30 20:37:51.812941 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:51.812871 2580 generic.go:356] "Generic (PLEG): container finished" podID="6ce76c71-8b23-49b4-a61c-9b14adea86c7" containerID="feed76ffb0a57996a6e070056d4f73660fcc8f2bf1b83847546edd0bba42dd54" exitCode=0 Oct 30 20:37:51.812941 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:51.812919 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n7lg7" event={"ID":"6ce76c71-8b23-49b4-a61c-9b14adea86c7","Type":"ContainerDied","Data":"feed76ffb0a57996a6e070056d4f73660fcc8f2bf1b83847546edd0bba42dd54"} Oct 30 20:37:52.820813 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:52.820493 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n7lg7" event={"ID":"6ce76c71-8b23-49b4-a61c-9b14adea86c7","Type":"ContainerStarted","Data":"c747d28e17321b395f6a91b28586fe1e3a04386eae2af143252468be48807edb"} Oct 30 20:37:52.823443 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:52.823421 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" event={"ID":"9224f6c0-bc3e-4eee-a78b-431e6c8f4705","Type":"ContainerStarted","Data":"c9f62a1f9da34745cf51e7d1d2fc642986e1c40c5544e5f84bbfc94e2a0533c3"} Oct 30 20:37:52.823546 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:52.823476 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" event={"ID":"9224f6c0-bc3e-4eee-a78b-431e6c8f4705","Type":"ContainerStarted","Data":"edf112e02776893327264b21a0f35e894045a6d4d14aa4c2a1df078b762df63c"} Oct 30 20:37:52.823546 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:52.823507 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" event={"ID":"9224f6c0-bc3e-4eee-a78b-431e6c8f4705","Type":"ContainerStarted","Data":"e9bc8f4f79bef6d92e373527c6d241cf67a22965669d0cbdf01da55bade96913"} Oct 30 20:37:52.823546 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:52.823536 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" event={"ID":"9224f6c0-bc3e-4eee-a78b-431e6c8f4705","Type":"ContainerStarted","Data":"6d9baa2bd0ff937cc7e88a6821e3a5eb517810128c9f05b4ac3fd9569d9a5776"} Oct 30 20:37:52.823636 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:52.823563 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" event={"ID":"9224f6c0-bc3e-4eee-a78b-431e6c8f4705","Type":"ContainerStarted","Data":"a249217c2dc19e6584ce05c3499ebf34240614f520807b7bdcda1258b22a53ce"} Oct 30 20:37:52.823636 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:52.823590 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" event={"ID":"9224f6c0-bc3e-4eee-a78b-431e6c8f4705","Type":"ContainerStarted","Data":"beec08de8f7e22aaaf77486227ef3b12245efe84ff3605ead7b321ee37a826e9"} Oct 30 20:37:52.839826 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:52.839785 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n7lg7" podStartSLOduration=8.750019859 podStartE2EDuration="18.839753083s" totalImagesPullingTime="10.089733224s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:52.83936811 +0000 UTC m=+31.846616420" watchObservedRunningTime="2026-10-30 20:37:52.839753083 +0000 UTC m=+31.847001389" Oct 30 20:37:53.263567 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:53.263492 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:53.263686 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:53.263629 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:53.263686 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:53.263683 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret podName:a99706d1-d012-4dd9-b46a-5ec139287351 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:09.263666346 +0000 UTC m=+48.270914648 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret") pod "global-pull-secret-syncer-5wp5n" (UID: "a99706d1-d012-4dd9-b46a-5ec139287351") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:37:53.364616 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:53.364549 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs\") pod \"network-metrics-daemon-48zsl\" (UID: \"51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249\") " pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:53.364751 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:53.364690 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:53.364751 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:53.364748 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs podName:51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:09.364731825 +0000 UTC m=+48.371980113 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs") pod "network-metrics-daemon-48zsl" (UID: "51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:37:53.465688 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:53.465658 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7456\" (UniqueName: \"kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456\") pod \"network-check-target-qs2dh\" (UID: \"0832230b-2d6a-40f2-8d5f-dc1da218c7b8\") " pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:37:53.465898 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:53.465796 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:37:53.465898 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:53.465811 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:37:53.465898 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:53.465820 2580 projected.go:196] Error preparing data for projected volume kube-api-access-v7456 for pod openshift-network-diagnostics/network-check-target-qs2dh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:53.465898 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:53.465869 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456 podName:0832230b-2d6a-40f2-8d5f-dc1da218c7b8 nodeName:}" failed. No retries permitted until 2026-10-30 20:38:09.465850913 +0000 UTC m=+48.473099204 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v7456" (UniqueName: "kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456") pod "network-check-target-qs2dh" (UID: "0832230b-2d6a-40f2-8d5f-dc1da218c7b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:37:53.473457 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:53.473438 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:53.473537 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:53.473441 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:53.473537 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:53.473527 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wp5n" podUID="a99706d1-d012-4dd9-b46a-5ec139287351" Oct 30 20:37:53.473620 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:53.473445 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:37:53.473620 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:53.473610 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48zsl" podUID="51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249" Oct 30 20:37:53.473721 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:53.473702 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs2dh" podUID="0832230b-2d6a-40f2-8d5f-dc1da218c7b8" Oct 30 20:37:54.830922 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:54.830725 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" event={"ID":"9224f6c0-bc3e-4eee-a78b-431e6c8f4705","Type":"ContainerStarted","Data":"d57b53f982c050e5bbe325b29c9186d6ac9b2c581f99e4e000008723aa580cd5"} Oct 30 20:37:55.473621 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:55.473590 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:37:55.473813 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:55.473626 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:55.473813 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:55.473695 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs2dh" podUID="0832230b-2d6a-40f2-8d5f-dc1da218c7b8" Oct 30 20:37:55.473813 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:55.473733 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:55.473978 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:55.473855 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48zsl" podUID="51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249" Oct 30 20:37:55.473978 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:55.473926 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wp5n" podUID="a99706d1-d012-4dd9-b46a-5ec139287351" Oct 30 20:37:56.839127 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:56.839102 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" event={"ID":"9224f6c0-bc3e-4eee-a78b-431e6c8f4705","Type":"ContainerStarted","Data":"9f36d9b0d02427e6789d23b699fc8de0a50ecfbdb4f0393140db25e1642090e7"} Oct 30 20:37:57.473632 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:57.473602 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:57.473838 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:57.473601 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:57.473838 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:57.473702 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wp5n" podUID="a99706d1-d012-4dd9-b46a-5ec139287351" Oct 30 20:37:57.473929 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:57.473833 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48zsl" podUID="51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249" Oct 30 20:37:57.473929 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:57.473606 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:37:57.474000 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:57.473924 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs2dh" podUID="0832230b-2d6a-40f2-8d5f-dc1da218c7b8" Oct 30 20:37:57.841929 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:57.841857 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:57.857346 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:57.857321 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:57.876341 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:57.874615 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" podStartSLOduration=9.396783097 podStartE2EDuration="23.874598732s" totalImagesPullingTime="14.477815635s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:37:57.873577282 +0000 UTC m=+36.880825589" watchObservedRunningTime="2026-10-30 20:37:57.874598732 +0000 UTC m=+36.881847043" Oct 30 20:37:58.555892 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:58.555827 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qs2dh"] Oct 30 20:37:58.556032 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:58.555977 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:37:58.556122 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:58.556088 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs2dh" podUID="0832230b-2d6a-40f2-8d5f-dc1da218c7b8" Oct 30 20:37:58.559987 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:58.559960 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-48zsl"] Oct 30 20:37:58.560116 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:58.560076 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:37:58.560223 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:58.560183 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48zsl" podUID="51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249" Oct 30 20:37:58.560707 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:58.560686 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5wp5n"] Oct 30 20:37:58.560808 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:58.560796 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:37:58.560883 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:37:58.560869 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wp5n" podUID="a99706d1-d012-4dd9-b46a-5ec139287351" Oct 30 20:37:58.844502 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:58.844422 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:58.844502 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:58.844503 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:37:58.859234 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:37:58.859210 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:00.473524 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:00.473501 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:38:00.473896 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:00.473501 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:38:00.473896 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:38:00.473597 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48zsl" podUID="51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249" Oct 30 20:38:00.473896 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:00.473503 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:38:00.473896 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:38:00.473666 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs2dh" podUID="0832230b-2d6a-40f2-8d5f-dc1da218c7b8" Oct 30 20:38:00.473896 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:38:00.473726 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wp5n" podUID="a99706d1-d012-4dd9-b46a-5ec139287351" Oct 30 20:38:01.464213 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:01.464027 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-grp6n" Oct 30 20:38:01.464424 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:01.464363 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:38:01.464859 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:01.464838 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-grp6n" Oct 30 20:38:02.473607 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:02.473579 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:38:02.473607 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:02.473598 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:38:02.474181 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:02.473661 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:38:02.474181 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:38:02.473754 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48zsl" podUID="51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249" Oct 30 20:38:02.474181 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:38:02.473857 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs2dh" podUID="0832230b-2d6a-40f2-8d5f-dc1da218c7b8" Oct 30 20:38:02.474181 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:38:02.473939 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wp5n" podUID="a99706d1-d012-4dd9-b46a-5ec139287351" Oct 30 20:38:04.474442 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.474404 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:38:04.474442 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.474441 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:38:04.474965 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.474500 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:38:04.474965 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:38:04.474510 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wp5n" podUID="a99706d1-d012-4dd9-b46a-5ec139287351" Oct 30 20:38:04.474965 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:38:04.474569 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qs2dh" podUID="0832230b-2d6a-40f2-8d5f-dc1da218c7b8" Oct 30 20:38:04.474965 ip-10-0-130-52 kubenswrapper[2580]: E1030 20:38:04.474648 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48zsl" podUID="51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249" Oct 30 20:38:04.711808 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.711758 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeReady" Oct 30 20:38:04.711928 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.711905 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 20:38:04.790958 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.790893 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9w8gw"] Oct 30 20:38:04.804017 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.803993 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9w8gw" Oct 30 20:38:04.804897 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.804795 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tvdg8"] Oct 30 20:38:04.814064 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.814042 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9w8gw"] Oct 30 20:38:04.814159 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.814148 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:04.815684 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.815668 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 20:38:04.815777 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.815721 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 20:38:04.820584 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.820564 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 20:38:04.820975 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.820953 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-69hq7\"" Oct 30 20:38:04.821089 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.821072 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 20:38:04.821493 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.821478 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rsx8b\"" Oct 30 20:38:04.824560 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.824544 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k4d59"] Oct 30 20:38:04.830756 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.830741 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:04.848183 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.848154 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7ns5\" (UniqueName: \"kubernetes.io/projected/a8e2605f-358c-4fd1-ae8a-524282c76f72-kube-api-access-s7ns5\") pod \"ingress-canary-9w8gw\" (UID: \"a8e2605f-358c-4fd1-ae8a-524282c76f72\") " pod="openshift-ingress-canary/ingress-canary-9w8gw" Oct 30 20:38:04.848322 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.848298 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8e2605f-358c-4fd1-ae8a-524282c76f72-cert\") pod \"ingress-canary-9w8gw\" (UID: \"a8e2605f-358c-4fd1-ae8a-524282c76f72\") " pod="openshift-ingress-canary/ingress-canary-9w8gw" Oct 30 20:38:04.848918 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.848887 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k4d59"] Oct 30 20:38:04.849719 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.849693 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tvdg8"] Oct 30 20:38:04.912281 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.912246 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 20:38:04.912281 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.912279 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 20:38:04.912466 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.912303 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 20:38:04.912466 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.912314 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 20:38:04.921545 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.921524 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qtwx7\"" Oct 30 20:38:04.921545 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.921532 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 20:38:04.948799 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.948759 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dbe199fd-1f79-4fa3-9136-21a58fa1202d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:04.948910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.948824 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7ns5\" (UniqueName: \"kubernetes.io/projected/a8e2605f-358c-4fd1-ae8a-524282c76f72-kube-api-access-s7ns5\") pod \"ingress-canary-9w8gw\" (UID: \"a8e2605f-358c-4fd1-ae8a-524282c76f72\") " pod="openshift-ingress-canary/ingress-canary-9w8gw" Oct 30 20:38:04.948910 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.948873 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slwwv\" (UniqueName: \"kubernetes.io/projected/dbe199fd-1f79-4fa3-9136-21a58fa1202d-kube-api-access-slwwv\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:04.949012 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.948930 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b66dabb7-8606-4898-99ca-954ffd602da2-config-volume\") pod \"dns-default-tvdg8\" (UID: \"b66dabb7-8606-4898-99ca-954ffd602da2\") " pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:04.949012 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.948953 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b66dabb7-8606-4898-99ca-954ffd602da2-metrics-tls\") pod \"dns-default-tvdg8\" (UID: \"b66dabb7-8606-4898-99ca-954ffd602da2\") " pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:04.949012 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.948976 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqs5f\" (UniqueName: \"kubernetes.io/projected/b66dabb7-8606-4898-99ca-954ffd602da2-kube-api-access-mqs5f\") pod \"dns-default-tvdg8\" (UID: \"b66dabb7-8606-4898-99ca-954ffd602da2\") " pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:04.949155 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.949043 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dbe199fd-1f79-4fa3-9136-21a58fa1202d-crio-socket\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:04.949155 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.949115 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8e2605f-358c-4fd1-ae8a-524282c76f72-cert\") pod \"ingress-canary-9w8gw\" (UID: \"a8e2605f-358c-4fd1-ae8a-524282c76f72\") " pod="openshift-ingress-canary/ingress-canary-9w8gw" Oct 30 20:38:04.949155 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.949143 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbe199fd-1f79-4fa3-9136-21a58fa1202d-data-volume\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:04.949259 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.949165 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dbe199fd-1f79-4fa3-9136-21a58fa1202d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:04.949259 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.949219 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b66dabb7-8606-4898-99ca-954ffd602da2-tmp-dir\") pod \"dns-default-tvdg8\" (UID: \"b66dabb7-8606-4898-99ca-954ffd602da2\") " pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:04.953015 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.952989 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8e2605f-358c-4fd1-ae8a-524282c76f72-cert\") pod \"ingress-canary-9w8gw\" (UID: \"a8e2605f-358c-4fd1-ae8a-524282c76f72\") " pod="openshift-ingress-canary/ingress-canary-9w8gw" Oct 30 20:38:04.959052 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:04.959029 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7ns5\" (UniqueName: \"kubernetes.io/projected/a8e2605f-358c-4fd1-ae8a-524282c76f72-kube-api-access-s7ns5\") pod \"ingress-canary-9w8gw\" (UID: \"a8e2605f-358c-4fd1-ae8a-524282c76f72\") " pod="openshift-ingress-canary/ingress-canary-9w8gw" Oct 30 20:38:05.050032 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.049967 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dbe199fd-1f79-4fa3-9136-21a58fa1202d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:05.050032 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.050014 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slwwv\" (UniqueName: \"kubernetes.io/projected/dbe199fd-1f79-4fa3-9136-21a58fa1202d-kube-api-access-slwwv\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:05.050228 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.050051 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b66dabb7-8606-4898-99ca-954ffd602da2-config-volume\") pod \"dns-default-tvdg8\" (UID: \"b66dabb7-8606-4898-99ca-954ffd602da2\") " pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:05.050228 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.050078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b66dabb7-8606-4898-99ca-954ffd602da2-metrics-tls\") pod \"dns-default-tvdg8\" (UID: \"b66dabb7-8606-4898-99ca-954ffd602da2\") " pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:05.050228 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.050106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqs5f\" (UniqueName: \"kubernetes.io/projected/b66dabb7-8606-4898-99ca-954ffd602da2-kube-api-access-mqs5f\") pod \"dns-default-tvdg8\" (UID: \"b66dabb7-8606-4898-99ca-954ffd602da2\") " pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:05.050228 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.050132 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dbe199fd-1f79-4fa3-9136-21a58fa1202d-crio-socket\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:05.050228 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.050157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbe199fd-1f79-4fa3-9136-21a58fa1202d-data-volume\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:05.050228 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.050181 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dbe199fd-1f79-4fa3-9136-21a58fa1202d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:05.050228 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.050207 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b66dabb7-8606-4898-99ca-954ffd602da2-tmp-dir\") pod \"dns-default-tvdg8\" (UID: \"b66dabb7-8606-4898-99ca-954ffd602da2\") " pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:05.050576 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.050444 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dbe199fd-1f79-4fa3-9136-21a58fa1202d-crio-socket\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:05.050576 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.050534 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbe199fd-1f79-4fa3-9136-21a58fa1202d-data-volume\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:05.050841 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.050823 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b66dabb7-8606-4898-99ca-954ffd602da2-config-volume\") pod \"dns-default-tvdg8\" (UID: \"b66dabb7-8606-4898-99ca-954ffd602da2\") " pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:05.050896 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.050832 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dbe199fd-1f79-4fa3-9136-21a58fa1202d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:05.050896 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.050853 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b66dabb7-8606-4898-99ca-954ffd602da2-tmp-dir\") pod \"dns-default-tvdg8\" (UID: \"b66dabb7-8606-4898-99ca-954ffd602da2\") " pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:05.052524 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.052503 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dbe199fd-1f79-4fa3-9136-21a58fa1202d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:05.052658 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.052641 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b66dabb7-8606-4898-99ca-954ffd602da2-metrics-tls\") pod \"dns-default-tvdg8\" (UID: \"b66dabb7-8606-4898-99ca-954ffd602da2\") " pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:05.063084 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.063054 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slwwv\" (UniqueName: \"kubernetes.io/projected/dbe199fd-1f79-4fa3-9136-21a58fa1202d-kube-api-access-slwwv\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:05.063256 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.063238 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqs5f\" (UniqueName: \"kubernetes.io/projected/b66dabb7-8606-4898-99ca-954ffd602da2-kube-api-access-mqs5f\") pod \"dns-default-tvdg8\" (UID: \"b66dabb7-8606-4898-99ca-954ffd602da2\") " pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:05.124862 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.124833 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9w8gw" Oct 30 20:38:05.134001 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.133976 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:05.155281 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.155258 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:05.478037 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.477830 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9w8gw"] Oct 30 20:38:05.478863 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.478843 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tvdg8"] Oct 30 20:38:05.479419 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:38:05.479398 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e2605f_358c_4fd1_ae8a_524282c76f72.slice/crio-b7cf93a51c0fd37a6cbae375e98663034a455e030f5815c51c03b85642ed1aae WatchSource:0}: Error finding container b7cf93a51c0fd37a6cbae375e98663034a455e030f5815c51c03b85642ed1aae: Status 404 returned error can't find the container with id b7cf93a51c0fd37a6cbae375e98663034a455e030f5815c51c03b85642ed1aae Oct 30 20:38:05.481402 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:38:05.481385 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66dabb7_8606_4898_99ca_954ffd602da2.slice/crio-d722d916e270501ded9f2d355bc57579a0a92010a113ca8ff4f2559a831fab2c WatchSource:0}: Error finding container d722d916e270501ded9f2d355bc57579a0a92010a113ca8ff4f2559a831fab2c: Status 404 returned error can't find the container with id d722d916e270501ded9f2d355bc57579a0a92010a113ca8ff4f2559a831fab2c Oct 30 20:38:05.487519 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.487421 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k4d59"] Oct 30 20:38:05.489788 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:38:05.489748 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe199fd_1f79_4fa3_9136_21a58fa1202d.slice/crio-2d8faca17500c54dbeeb859698812b94b33e516f72c0b1570ce36af28eba44c2 WatchSource:0}: Error finding container 2d8faca17500c54dbeeb859698812b94b33e516f72c0b1570ce36af28eba44c2: Status 404 returned error can't find the container with id 2d8faca17500c54dbeeb859698812b94b33e516f72c0b1570ce36af28eba44c2 Oct 30 20:38:05.861313 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.861245 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9w8gw" event={"ID":"a8e2605f-358c-4fd1-ae8a-524282c76f72","Type":"ContainerStarted","Data":"b7cf93a51c0fd37a6cbae375e98663034a455e030f5815c51c03b85642ed1aae"} Oct 30 20:38:05.862683 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.862654 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k4d59" event={"ID":"dbe199fd-1f79-4fa3-9136-21a58fa1202d","Type":"ContainerStarted","Data":"63f8c5864703da59738d9e622c4c0064e3b458cfec1dd19e32e5ef81de493661"} Oct 30 20:38:05.862807 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.862697 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k4d59" event={"ID":"dbe199fd-1f79-4fa3-9136-21a58fa1202d","Type":"ContainerStarted","Data":"2d8faca17500c54dbeeb859698812b94b33e516f72c0b1570ce36af28eba44c2"} Oct 30 20:38:05.863671 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:05.863650 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tvdg8" event={"ID":"b66dabb7-8606-4898-99ca-954ffd602da2","Type":"ContainerStarted","Data":"d722d916e270501ded9f2d355bc57579a0a92010a113ca8ff4f2559a831fab2c"} Oct 30 20:38:06.474662 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:06.473493 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:38:06.474662 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:06.474124 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:38:06.474662 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:06.474158 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:38:06.479798 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:06.479761 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4pvnz\"" Oct 30 20:38:06.480292 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:06.480274 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 20:38:06.480385 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:06.480305 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 20:38:06.480385 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:06.480371 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 20:38:06.480501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:06.480418 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zljsp\"" Oct 30 20:38:06.480501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:06.480434 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jrl7k\"" Oct 30 20:38:06.480501 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:06.480485 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 20:38:06.868863 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:06.868785 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k4d59" event={"ID":"dbe199fd-1f79-4fa3-9136-21a58fa1202d","Type":"ContainerStarted","Data":"68adb81388de59f89e06c7b9da38c1b2d1fc4785c91695b2b516bc2bc6f014c9"} Oct 30 20:38:08.877441 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:08.877411 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tvdg8" event={"ID":"b66dabb7-8606-4898-99ca-954ffd602da2","Type":"ContainerStarted","Data":"47a37b45354adddfcbde247510577389725a75e4a6f39a492bbff6423f3f9a80"} Oct 30 20:38:08.877830 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:08.877494 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tvdg8" event={"ID":"b66dabb7-8606-4898-99ca-954ffd602da2","Type":"ContainerStarted","Data":"aa9c5f881523aaa45dce05aa0fa506b385f1202e5dadf579e9c6d24eb66b2163"} Oct 30 20:38:08.877830 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:08.877577 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:08.900289 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:08.900239 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-tvdg8" podStartSLOduration=2.627421858 podStartE2EDuration="4.90022539s" totalImagesPullingTime="2.272803532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:08.899018764 +0000 UTC m=+47.906267074" watchObservedRunningTime="2026-10-30 20:38:08.90022539 +0000 UTC m=+47.907473699" Oct 30 20:38:09.292081 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:09.292046 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:38:09.295218 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:09.295197 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:38:09.392624 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:09.392591 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs\") pod \"network-metrics-daemon-48zsl\" (UID: \"51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249\") " pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:38:09.394696 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:09.394675 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs\") pod \"network-metrics-daemon-48zsl\" (UID: \"51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249\") " pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:38:09.493302 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:09.493276 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7456\" (UniqueName: \"kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456\") pod \"network-check-target-qs2dh\" (UID: \"0832230b-2d6a-40f2-8d5f-dc1da218c7b8\") " pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:38:09.495539 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:09.495520 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7456\" (UniqueName: \"kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456\") pod \"network-check-target-qs2dh\" (UID: \"0832230b-2d6a-40f2-8d5f-dc1da218c7b8\") " pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:38:09.532409 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:09.532385 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:38:09.537482 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:09.537466 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:38:09.549497 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:09.549451 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:38:09.819269 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:09.819202 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-48zsl"] Oct 30 20:38:09.822903 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:38:09.822874 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51eb0ba7_2dfd_471f_98e1_5dcdb1ab2249.slice/crio-c107fdaadcf3cd5d859b5fa6bb4a7c89781203673077a42f0462fc84ee9459f4 WatchSource:0}: Error finding container c107fdaadcf3cd5d859b5fa6bb4a7c89781203673077a42f0462fc84ee9459f4: Status 404 returned error can't find the container with id c107fdaadcf3cd5d859b5fa6bb4a7c89781203673077a42f0462fc84ee9459f4 Oct 30 20:38:09.858870 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:09.858848 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5wp5n"] Oct 30 20:38:09.861365 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:38:09.861345 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99706d1_d012_4dd9_b46a_5ec139287351.slice/crio-004bdc84b9dfb4f483b8235c30cc031935a4f53a22787b2b8a231cf5b72e1395 WatchSource:0}: Error finding container 004bdc84b9dfb4f483b8235c30cc031935a4f53a22787b2b8a231cf5b72e1395: Status 404 returned error can't find the container with id 004bdc84b9dfb4f483b8235c30cc031935a4f53a22787b2b8a231cf5b72e1395 Oct 30 20:38:09.882615 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:09.882589 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9w8gw" event={"ID":"a8e2605f-358c-4fd1-ae8a-524282c76f72","Type":"ContainerStarted","Data":"cc764e0a14f6ab78183ee0bfc5596ddc955b8dc664adb966e8487ae90ff28e17"} Oct 30 20:38:09.883732 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:09.883711 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5wp5n" event={"ID":"a99706d1-d012-4dd9-b46a-5ec139287351","Type":"ContainerStarted","Data":"004bdc84b9dfb4f483b8235c30cc031935a4f53a22787b2b8a231cf5b72e1395"} Oct 30 20:38:09.885490 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:09.885467 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k4d59" event={"ID":"dbe199fd-1f79-4fa3-9136-21a58fa1202d","Type":"ContainerStarted","Data":"efa28c6a6bb11fe6e2b2efa8cf1236ec8f21a13f4c737ed047d0e504d4029849"} Oct 30 20:38:09.886559 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:09.886537 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-48zsl" event={"ID":"51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249","Type":"ContainerStarted","Data":"c107fdaadcf3cd5d859b5fa6bb4a7c89781203673077a42f0462fc84ee9459f4"} Oct 30 20:38:09.888164 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:09.888148 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qs2dh"] Oct 30 20:38:09.890178 ip-10-0-130-52 kubenswrapper[2580]: W1030 20:38:09.890150 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0832230b_2d6a_40f2_8d5f_dc1da218c7b8.slice/crio-3d0a8452e963c37b0d11205dc2df943eeba902182d469728a1f8471f255da9e9 WatchSource:0}: Error finding container 3d0a8452e963c37b0d11205dc2df943eeba902182d469728a1f8471f255da9e9: Status 404 returned error can't find the container with id 3d0a8452e963c37b0d11205dc2df943eeba902182d469728a1f8471f255da9e9 Oct 30 20:38:09.901191 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:09.901152 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9w8gw" podStartSLOduration=2.542648941 podStartE2EDuration="5.9011395s" totalImagesPullingTime="3.358490559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:09.900692984 +0000 UTC m=+48.907941293" watchObservedRunningTime="2026-10-30 20:38:09.9011395 +0000 UTC m=+48.908387807" Oct 30 20:38:09.933079 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:09.933028 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k4d59" podStartSLOduration=2.774498876 podStartE2EDuration="5.933017136s" totalImagesPullingTime="3.15851826s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:38:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:09.932353137 +0000 UTC m=+48.939601448" watchObservedRunningTime="2026-10-30 20:38:09.933017136 +0000 UTC m=+48.940265446" Oct 30 20:38:10.892126 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:10.892090 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qs2dh" event={"ID":"0832230b-2d6a-40f2-8d5f-dc1da218c7b8","Type":"ContainerStarted","Data":"3d0a8452e963c37b0d11205dc2df943eeba902182d469728a1f8471f255da9e9"} Oct 30 20:38:11.898463 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:11.897919 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-48zsl" event={"ID":"51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249","Type":"ContainerStarted","Data":"4fe0d5e8c62439934000de89972c33a0c9bfbd590b55325b8aada9dab78012ad"} Oct 30 20:38:11.898463 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:11.897991 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-48zsl" event={"ID":"51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249","Type":"ContainerStarted","Data":"3913eacf834255ad535c5cb3409ebdec7df0b967f56e1e3e91fb0a34e9fef013"} Oct 30 20:38:11.915436 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:11.915380 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-48zsl" podStartSLOduration=36.240263356 podStartE2EDuration="37.915362248s" totalImagesPullingTime="1.675098892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:11.914275475 +0000 UTC m=+50.921523787" watchObservedRunningTime="2026-10-30 20:38:11.915362248 +0000 UTC m=+50.922610559" Oct 30 20:38:12.107629 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:12.107519 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grp6n_4c57ecae-89cb-4901-8f9e-e4586771f5d8/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:12.193742 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:12.193654 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grp6n_4c57ecae-89cb-4901-8f9e-e4586771f5d8/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:12.293261 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:12.293230 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-grp6n_4c57ecae-89cb-4901-8f9e-e4586771f5d8/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:13.915592 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:13.915546 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qs2dh" event={"ID":"0832230b-2d6a-40f2-8d5f-dc1da218c7b8","Type":"ContainerStarted","Data":"4163b0cef4b045d34b82c7a16c63eebe8e17b97afd0b29bf5256605c1e880206"} Oct 30 20:38:13.916054 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:13.915688 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:38:13.936761 ip-10-0-130-52 kubenswrapper[2580]: I1030 20:38:13.936708 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qs2dh" podStartSLOduration=36.420185173 podStartE2EDuration="39.936670045s" totalImagesPullingTime="3.516484872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:13.935688263 +0000 UTC m=+52.942936566" watchObservedRunningTime="2026-10-30 20:38:13.936670045 +0000 UTC m=+52.943918395" Oct 30 20:38:14.756161 ip-10-0-130-52 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 20:38:14.774176 ip-10-0-130-52 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 20:38:14.774341 ip-10-0-130-52 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 20:38:14.774509 ip-10-0-130-52 systemd[1]: kubelet.service: Consumed 2.772s CPU time. Oct 30 20:38:14.790478 ip-10-0-130-52 systemd[1]: Starting Kubernetes Kubelet... Oct 30 20:38:14.851728 ip-10-0-130-52 kubenswrapper[4848]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:38:14.851728 ip-10-0-130-52 kubenswrapper[4848]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 20:38:14.851728 ip-10-0-130-52 kubenswrapper[4848]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:38:14.851728 ip-10-0-130-52 kubenswrapper[4848]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:38:14.853809 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853717 4848 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:14.853809 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853808 4848 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:14.853809 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853814 4848 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:14.853907 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853817 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:14.853907 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853821 4848 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:14.853907 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853824 4848 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:14.853907 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853827 4848 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:14.853907 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853830 4848 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:14.853907 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853833 4848 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:14.853907 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853836 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:14.853907 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853839 4848 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:14.853907 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853842 4848 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:14.853907 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853844 4848 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:14.853907 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853847 4848 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:14.853907 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853850 4848 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:14.853907 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853852 4848 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:14.853907 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853855 4848 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:14.853907 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853857 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:14.853907 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853860 4848 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:14.853907 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853862 4848 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:14.853907 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853865 4848 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:14.853907 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853868 4848 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:14.853907 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853871 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:14.854425 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853873 4848 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:14.854425 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853876 4848 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:14.854425 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853878 4848 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:14.854425 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853893 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:14.854425 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853898 4848 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:14.854425 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853902 4848 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:14.854425 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853905 4848 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:14.854425 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853908 4848 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:14.854425 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853911 4848 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:14.854425 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853914 4848 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:14.854425 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853917 4848 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:14.854425 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853920 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:14.854425 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853923 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:14.854425 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853925 4848 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:14.854425 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853928 4848 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:14.854425 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853931 4848 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:14.854425 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853934 4848 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:14.854425 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853937 4848 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:14.854425 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853939 4848 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:14.854425 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853942 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:14.855072 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853944 4848 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:14.855072 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853947 4848 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:14.855072 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853949 4848 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:14.855072 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853952 4848 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:14.855072 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853954 4848 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:14.855072 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853957 4848 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:14.855072 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853959 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:14.855072 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853962 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:14.855072 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853965 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:14.855072 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853967 4848 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:14.855072 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853970 4848 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:14.855072 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853973 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:14.855072 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853975 4848 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:14.855072 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853978 4848 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:14.855072 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853980 4848 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:14.855072 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853983 4848 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:14.855072 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853991 4848 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:14.855072 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853994 4848 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:14.855072 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853997 4848 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:14.855540 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.853999 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:14.855540 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854002 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:14.855540 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854004 4848 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:14.855540 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854007 4848 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:14.855540 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854009 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:14.855540 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854012 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:14.855540 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854015 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:14.855540 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854017 4848 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:14.855540 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854020 4848 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:14.855540 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854022 4848 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:14.855540 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854025 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:14.855540 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854027 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:14.855540 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854030 4848 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:14.855540 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854033 4848 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:14.855540 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854035 4848 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:14.855540 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854038 4848 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:14.855540 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854040 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:14.855540 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854043 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:14.855540 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854045 4848 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:14.856046 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854048 4848 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:14.856046 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854050 4848 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:14.856046 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854054 4848 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:14.856046 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854057 4848 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:14.856046 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854060 4848 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:14.856046 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854063 4848 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:14.856046 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854066 4848 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:14.856046 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854069 4848 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:14.856046 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854071 4848 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:14.856046 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854074 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:14.856046 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854076 4848 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:14.856046 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854085 4848 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:14.856046 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854090 4848 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:14.856046 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854094 4848 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:14.856046 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854097 4848 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:14.856046 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854100 4848 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:14.856046 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854103 4848 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:14.856046 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854105 4848 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:14.856046 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854108 4848 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:14.856578 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854110 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:14.856578 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854113 4848 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:14.856578 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854115 4848 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:14.856578 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854118 4848 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:14.856578 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854120 4848 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:14.856578 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854123 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:14.856578 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854891 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:14.856578 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854898 4848 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:14.856578 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854900 4848 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:14.856578 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854903 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:14.856578 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854906 4848 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:14.856578 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854909 4848 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:14.856578 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854912 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:14.856578 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854915 4848 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:14.856578 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854918 4848 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:14.856578 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854920 4848 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:14.856578 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854923 4848 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:14.856578 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854928 4848 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:14.856578 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854931 4848 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:14.856578 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854933 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:14.857106 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854936 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:14.857106 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854939 4848 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:14.857106 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854941 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:14.857106 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854944 4848 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:14.857106 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854947 4848 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:14.857106 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854950 4848 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:14.857106 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854953 4848 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:14.857106 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854955 4848 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:14.857106 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854958 4848 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:14.857106 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854961 4848 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:14.857106 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854964 4848 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:14.857106 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854967 4848 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:14.857106 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854969 4848 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:14.857106 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854972 4848 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:14.857106 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854974 4848 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:14.857106 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854977 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:14.857106 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854982 4848 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:14.857106 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854987 4848 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:14.857106 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854990 4848 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:14.857630 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854993 4848 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:14.857630 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854996 4848 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:14.857630 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.854998 4848 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:14.857630 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855001 4848 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:14.857630 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855003 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:14.857630 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855006 4848 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:14.857630 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855008 4848 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:14.857630 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855011 4848 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:14.857630 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855014 4848 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:14.857630 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855016 4848 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:14.857630 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855021 4848 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:14.857630 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855024 4848 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:14.857630 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855027 4848 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:14.857630 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855029 4848 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:14.857630 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855032 4848 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:14.857630 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855034 4848 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:14.857630 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855037 4848 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:14.857630 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855040 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:14.857630 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855043 4848 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:14.857630 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855046 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:14.858160 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855048 4848 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:14.858160 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855051 4848 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:14.858160 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855054 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:14.858160 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855056 4848 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:14.858160 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855059 4848 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:14.858160 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855061 4848 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:14.858160 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855064 4848 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:14.858160 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855067 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:14.858160 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855069 4848 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:14.858160 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855072 4848 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:14.858160 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855075 4848 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:14.858160 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855077 4848 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:14.858160 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855080 4848 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:14.858160 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855083 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:14.858160 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855085 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:14.858160 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855089 4848 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:14.858160 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855091 4848 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:14.858160 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855094 4848 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:14.858160 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855096 4848 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:14.858160 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855099 4848 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:14.858659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855101 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:14.858659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855104 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:14.858659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855108 4848 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:14.858659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855111 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:14.858659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855113 4848 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:14.858659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855116 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:14.858659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855118 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:14.858659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855121 4848 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:14.858659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855125 4848 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:14.858659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855128 4848 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:14.858659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855131 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:14.858659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855134 4848 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:14.858659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855138 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:14.858659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855141 4848 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:14.858659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855144 4848 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:14.858659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855147 4848 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:14.858659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855149 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:14.858659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855153 4848 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:14.858659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855155 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:14.859164 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855158 4848 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:14.859164 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855160 4848 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:14.859164 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855163 4848 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:14.859164 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855165 4848 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:14.859164 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855168 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:14.859164 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855171 4848 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:14.859164 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855174 4848 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:14.859164 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855176 4848 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:14.859164 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855179 4848 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:14.859164 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855181 4848 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:14.859164 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855184 4848 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:14.859164 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855187 4848 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:14.859164 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855190 4848 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:14.859164 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855193 4848 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:14.859164 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855648 4848 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:14.859164 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855653 4848 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:14.859164 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855657 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:14.859164 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855659 4848 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:14.859164 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855662 4848 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:14.859164 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855665 4848 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:14.859702 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855667 4848 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:14.859702 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855670 4848 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:14.859702 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855673 4848 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:14.859702 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855675 4848 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:14.859702 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855678 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:14.859702 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855682 4848 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:14.859702 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855684 4848 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:14.859702 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855687 4848 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:14.859702 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855690 4848 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:14.859702 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855693 4848 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:14.859702 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855695 4848 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:14.859702 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855698 4848 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:14.859702 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855701 4848 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:14.859702 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855703 4848 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:14.859702 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855706 4848 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:14.859702 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855709 4848 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:14.859702 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855711 4848 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:14.859702 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855714 4848 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:14.859702 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855717 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:14.860192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855720 4848 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:14.860192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855722 4848 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:14.860192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855725 4848 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:14.860192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855728 4848 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:14.860192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855730 4848 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:14.860192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855733 4848 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:14.860192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855735 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:14.860192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855738 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:14.860192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855740 4848 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:14.860192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855743 4848 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:14.860192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855745 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:14.860192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855748 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:14.860192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855751 4848 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:14.860192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855753 4848 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:14.860192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855756 4848 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:14.860192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855759 4848 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:14.860192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855779 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:14.860192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855783 4848 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:14.860192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855786 4848 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:14.860192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855789 4848 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:14.860693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855792 4848 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:14.860693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855795 4848 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:14.860693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855797 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:14.860693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855800 4848 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:14.860693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855802 4848 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:14.860693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855805 4848 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:14.860693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855807 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:14.860693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855810 4848 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:14.860693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855813 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:14.860693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855815 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:14.860693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855818 4848 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:14.860693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855821 4848 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:14.860693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855823 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:14.860693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855826 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:14.860693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855828 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:14.860693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855832 4848 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:14.860693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855836 4848 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:14.860693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855839 4848 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:14.860693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855841 4848 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:14.861493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855844 4848 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:14.861493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855846 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:14.861493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855850 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:14.861493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855853 4848 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:14.861493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855855 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:14.861493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855858 4848 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:14.861493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855861 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:14.861493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855863 4848 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:14.861493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855866 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:14.861493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855868 4848 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:14.861493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855873 4848 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:14.861493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855876 4848 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:14.861493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855878 4848 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:14.861493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855881 4848 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:14.861493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855884 4848 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:14.861493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855887 4848 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:14.861493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855889 4848 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:14.861493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855892 4848 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:14.861493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855894 4848 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:14.861493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855897 4848 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:14.862287 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855899 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:14.862287 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855903 4848 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:14.862287 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855907 4848 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:14.862287 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855910 4848 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:14.862287 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855913 4848 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:14.862287 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855916 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:14.862287 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855918 4848 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:14.862287 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855921 4848 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:14.862287 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855924 4848 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:14.862287 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855927 4848 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:14.862287 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855929 4848 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:14.862287 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855932 4848 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:14.862287 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855935 4848 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:14.862287 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855937 4848 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:14.862287 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855940 4848 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:14.862287 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855942 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:14.862287 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855945 4848 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:14.862287 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855947 4848 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:14.862287 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855950 4848 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855953 4848 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855956 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.855959 4848 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856052 4848 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856059 4848 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856066 4848 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856070 4848 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856075 4848 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856079 4848 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856083 4848 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856087 4848 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856090 4848 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856093 4848 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856097 4848 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856100 4848 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856103 4848 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856106 4848 flags.go:64] FLAG: --cgroup-root="" Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856109 4848 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856113 4848 flags.go:64] FLAG: --client-ca-file="" Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856116 4848 flags.go:64] FLAG: --cloud-provider="external" Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856119 4848 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856123 4848 flags.go:64] FLAG: --cluster-domain="" Oct 30 20:38:14.862758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856126 4848 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856129 4848 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856132 4848 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856136 4848 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856139 4848 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856142 4848 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856145 4848 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856149 4848 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856152 4848 flags.go:64] FLAG: --contention-profiling="false" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856155 4848 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856158 4848 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856161 4848 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856164 4848 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856170 4848 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856173 4848 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856176 4848 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856179 4848 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856182 4848 flags.go:64] FLAG: --enable-server="true" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856186 4848 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856190 4848 flags.go:64] FLAG: --event-burst="100" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856194 4848 flags.go:64] FLAG: --event-qps="50" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856196 4848 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856199 4848 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856202 4848 flags.go:64] FLAG: --eviction-hard="" Oct 30 20:38:14.863343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856207 4848 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856210 4848 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856213 4848 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856216 4848 flags.go:64] FLAG: --eviction-soft="" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856219 4848 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856222 4848 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856225 4848 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856228 4848 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856232 4848 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856235 4848 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856238 4848 flags.go:64] FLAG: --feature-gates="" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856242 4848 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856245 4848 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856248 4848 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856259 4848 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856263 4848 flags.go:64] FLAG: --healthz-port="10248" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856266 4848 flags.go:64] FLAG: --help="false" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856269 4848 flags.go:64] FLAG: --hostname-override="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856272 4848 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856275 4848 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856278 4848 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856282 4848 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856286 4848 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 20:38:14.863964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856289 4848 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856291 4848 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856294 4848 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856297 4848 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856301 4848 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856304 4848 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856307 4848 flags.go:64] FLAG: --kube-reserved="" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856310 4848 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856313 4848 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856316 4848 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856319 4848 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856322 4848 flags.go:64] FLAG: --lock-file="" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856324 4848 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856327 4848 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856330 4848 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856336 4848 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856339 4848 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856342 4848 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856345 4848 flags.go:64] FLAG: --logging-format="text" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856348 4848 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856351 4848 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856354 4848 flags.go:64] FLAG: --manifest-url="" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856357 4848 flags.go:64] FLAG: --manifest-url-header="" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856361 4848 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856364 4848 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 20:38:14.864632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856368 4848 flags.go:64] FLAG: --max-pods="110" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856371 4848 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856374 4848 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856377 4848 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856380 4848 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856385 4848 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856388 4848 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856391 4848 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856398 4848 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856401 4848 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856404 4848 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856407 4848 flags.go:64] FLAG: --pod-cidr="" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856413 4848 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856416 4848 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856420 4848 flags.go:64] FLAG: --pods-per-core="0" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856423 4848 flags.go:64] FLAG: --port="10250" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856425 4848 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856428 4848 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-01592fd9ea0c03c56" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856431 4848 flags.go:64] FLAG: --qos-reserved="" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856434 4848 flags.go:64] FLAG: --read-only-port="10255" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856438 4848 flags.go:64] FLAG: --register-node="true" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856440 4848 flags.go:64] FLAG: --register-with-taints="" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856444 4848 flags.go:64] FLAG: --registry-burst="10" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856447 4848 flags.go:64] FLAG: --registry-qps="5" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856450 4848 flags.go:64] FLAG: --reserved-cpus="" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856453 4848 flags.go:64] FLAG: --reserved-memory="" Oct 30 20:38:14.865373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856457 4848 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856460 4848 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856465 4848 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856468 4848 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856471 4848 flags.go:64] FLAG: --runonce="false" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856474 4848 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856477 4848 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856480 4848 flags.go:64] FLAG: --seccomp-default="false" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856483 4848 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856486 4848 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856489 4848 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856494 4848 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856497 4848 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856500 4848 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856503 4848 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856506 4848 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856509 4848 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856512 4848 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856515 4848 flags.go:64] FLAG: --system-cgroups="" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856518 4848 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856524 4848 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856527 4848 flags.go:64] FLAG: --tls-cert-file="" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856529 4848 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856533 4848 flags.go:64] FLAG: --tls-min-version="" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856536 4848 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 20:38:14.866077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856539 4848 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856542 4848 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856545 4848 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856548 4848 flags.go:64] FLAG: --v="2" Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856552 4848 flags.go:64] FLAG: --version="false" Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856556 4848 flags.go:64] FLAG: --vmodule="" Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856560 4848 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.856563 4848 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856689 4848 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856695 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856698 4848 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856701 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856703 4848 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856706 4848 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856708 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856711 4848 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856714 4848 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856717 4848 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856720 4848 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856723 4848 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856726 4848 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:14.866875 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856729 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:14.867514 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856732 4848 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:14.867514 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856734 4848 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:14.867514 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856737 4848 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:14.867514 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856739 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:14.867514 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856742 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:14.867514 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856745 4848 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:14.867514 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856748 4848 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:14.867514 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856751 4848 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:14.867514 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856753 4848 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:14.867514 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856756 4848 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:14.867514 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856758 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:14.867514 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856761 4848 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:14.867514 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856780 4848 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:14.867514 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856783 4848 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:14.867514 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856786 4848 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:14.867514 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856789 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:14.867514 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856791 4848 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:14.867514 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856794 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:14.867514 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856797 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:14.867514 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856802 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:14.868232 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856805 4848 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:14.868232 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856808 4848 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:14.868232 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856810 4848 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:14.868232 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856813 4848 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:14.868232 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856816 4848 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:14.868232 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856819 4848 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:14.868232 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856823 4848 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:14.868232 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856826 4848 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:14.868232 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856830 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:14.868232 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856833 4848 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:14.868232 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856836 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:14.868232 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856839 4848 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:14.868232 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856842 4848 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:14.868232 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856844 4848 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:14.868232 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856847 4848 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:14.868232 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856850 4848 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:14.868232 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856852 4848 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:14.868232 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856855 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:14.868232 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856858 4848 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:14.869271 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856861 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:14.869271 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856864 4848 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:14.869271 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856867 4848 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:14.869271 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856869 4848 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:14.869271 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856872 4848 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:14.869271 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856875 4848 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:14.869271 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856877 4848 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:14.869271 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856880 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:14.869271 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856882 4848 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:14.869271 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856885 4848 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:14.869271 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856887 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:14.869271 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856890 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:14.869271 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856894 4848 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:14.869271 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856897 4848 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:14.869271 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856899 4848 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:14.869271 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856902 4848 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:14.869271 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856904 4848 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:14.869271 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856907 4848 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:14.869271 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856910 4848 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:14.869903 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856912 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:14.869903 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856915 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:14.869903 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856919 4848 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:14.869903 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856921 4848 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:14.869903 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856924 4848 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:14.869903 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856927 4848 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:14.869903 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856930 4848 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:14.869903 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856933 4848 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:14.869903 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856935 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:14.869903 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856938 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:14.869903 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856941 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:14.869903 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856943 4848 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:14.869903 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856946 4848 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:14.869903 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856949 4848 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:14.869903 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856952 4848 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:14.869903 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856954 4848 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:14.869903 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856957 4848 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:14.869903 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856959 4848 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:14.869903 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856962 4848 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:14.869903 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856965 4848 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:14.870474 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856967 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:14.870474 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856970 4848 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:14.870474 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856972 4848 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:14.870474 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856975 4848 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:14.870474 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856978 4848 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:14.870474 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856983 4848 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:14.870474 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856986 4848 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:14.870474 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856989 4848 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:14.870474 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856991 4848 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:14.870474 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856994 4848 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:14.870474 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856997 4848 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:14.870474 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.856999 4848 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:14.870474 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857002 4848 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:14.870474 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857004 4848 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:14.870474 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.857032 4848 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:38:14.870474 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857067 4848 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:14.870996 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857071 4848 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:14.870996 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857074 4848 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:14.870996 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857077 4848 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:14.870996 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857080 4848 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:14.870996 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857083 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:14.870996 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857085 4848 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:14.870996 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857088 4848 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:14.870996 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857090 4848 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:14.870996 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857093 4848 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:14.870996 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857097 4848 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:14.870996 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857101 4848 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:14.870996 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857104 4848 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:14.870996 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857107 4848 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:14.870996 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857110 4848 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:14.870996 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857113 4848 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:14.870996 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857115 4848 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:14.870996 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857118 4848 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:14.870996 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857120 4848 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:14.870996 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857123 4848 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:14.871513 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857126 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:14.871513 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857128 4848 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:14.871513 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857132 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:14.871513 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857135 4848 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:14.871513 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857137 4848 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:14.871513 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857140 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:14.871513 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857142 4848 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:14.871513 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857145 4848 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:14.871513 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857148 4848 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:14.871513 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857150 4848 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:14.871513 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857153 4848 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:14.871513 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857156 4848 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:14.871513 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857159 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:14.871513 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857162 4848 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:14.871513 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857165 4848 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:14.871513 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857167 4848 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:14.871513 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857170 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:14.871513 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857172 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:14.871513 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857175 4848 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:14.871513 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857178 4848 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:14.872097 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857181 4848 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:14.872097 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857184 4848 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:14.872097 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857186 4848 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:14.872097 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857189 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:14.872097 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857191 4848 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:14.872097 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857194 4848 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:14.872097 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857197 4848 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:14.872097 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857200 4848 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:14.872097 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857202 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:14.872097 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857205 4848 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:14.872097 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857207 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:14.872097 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857210 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:14.872097 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857212 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:14.872097 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857215 4848 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:14.872097 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857222 4848 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:14.872097 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857225 4848 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:14.872097 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857228 4848 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:14.872097 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857230 4848 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:14.872097 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857233 4848 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:14.872097 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857237 4848 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:14.872644 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857241 4848 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:14.872644 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857244 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:14.872644 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857246 4848 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:14.872644 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857249 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:14.872644 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857253 4848 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:14.872644 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857255 4848 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:14.872644 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857258 4848 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:14.872644 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857261 4848 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:14.872644 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857263 4848 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:14.872644 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857266 4848 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:14.872644 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857268 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:14.872644 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857271 4848 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:14.872644 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857274 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:14.872644 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857277 4848 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:14.872644 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857279 4848 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:14.872644 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857282 4848 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:14.872644 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857284 4848 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:14.872644 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857287 4848 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:14.872644 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857290 4848 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:14.873136 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857293 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:14.873136 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857295 4848 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:14.873136 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857298 4848 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:14.873136 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857300 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:14.873136 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857303 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:14.873136 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857306 4848 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:14.873136 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857309 4848 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:14.873136 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857313 4848 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:14.873136 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857315 4848 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:14.873136 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857318 4848 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:14.873136 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857320 4848 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:14.873136 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857323 4848 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:14.873136 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857325 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:14.873136 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857328 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:14.873136 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857331 4848 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:14.873136 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857333 4848 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:14.873136 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857336 4848 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:14.873136 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857340 4848 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:14.873136 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857342 4848 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:14.873659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857345 4848 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:14.873659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857347 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:14.873659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857351 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:14.873659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857353 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:14.873659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857356 4848 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:14.873659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857358 4848 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:14.873659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857361 4848 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:14.873659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.857364 4848 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:14.873659 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.861645 4848 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 20:38:14.873659 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.861695 4848 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 20:38:14.873659 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.861830 4848 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 20:38:14.873659 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.861968 4848 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 20:38:14.873659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862043 4848 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:14.873659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862050 4848 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:14.873659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862053 4848 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:14.873659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862056 4848 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:14.873659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862059 4848 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:14.873659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862061 4848 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:14.873659 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862065 4848 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:14.874354 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862068 4848 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:14.874354 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862070 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:14.874354 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862073 4848 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:14.874354 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862076 4848 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:14.874354 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862078 4848 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:14.874354 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862081 4848 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:14.874354 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862084 4848 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:14.874354 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862087 4848 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:14.874354 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862089 4848 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:14.874354 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862092 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:14.874354 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862094 4848 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:14.874354 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862100 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:14.874354 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862103 4848 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:14.874354 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862106 4848 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:14.874354 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862109 4848 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:14.874354 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862112 4848 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:14.874354 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862115 4848 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:14.874354 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862117 4848 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:14.874354 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862120 4848 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:14.874354 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862123 4848 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:14.874890 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862125 4848 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:14.874890 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862128 4848 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:14.874890 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862130 4848 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:14.874890 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862133 4848 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:14.874890 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862135 4848 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:14.874890 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862138 4848 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:14.874890 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862141 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:14.874890 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862143 4848 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:14.874890 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862148 4848 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:14.874890 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862155 4848 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:14.874890 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862158 4848 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:14.874890 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862161 4848 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:14.874890 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862163 4848 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:14.874890 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862166 4848 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:14.874890 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862169 4848 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:14.874890 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862171 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:14.874890 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862175 4848 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:14.874890 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862177 4848 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:14.874890 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862180 4848 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:14.875348 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862183 4848 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:14.875348 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862185 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:14.875348 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862188 4848 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:14.875348 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862193 4848 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:14.875348 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862196 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:14.875348 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862200 4848 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:14.875348 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862203 4848 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:14.875348 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862207 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:14.875348 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862210 4848 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:14.875348 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862213 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:14.875348 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862217 4848 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:14.875348 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862220 4848 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:14.875348 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862223 4848 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:14.875348 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862226 4848 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:14.875348 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862228 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:14.875348 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862231 4848 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:14.875348 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862234 4848 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:14.875348 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862236 4848 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:14.875348 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862239 4848 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:14.875829 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862241 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:14.875829 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862244 4848 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:14.875829 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862247 4848 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:14.875829 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862249 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:14.875829 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862252 4848 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:14.875829 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862254 4848 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:14.875829 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862257 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:14.875829 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862260 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:14.875829 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862262 4848 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:14.875829 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862265 4848 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:14.875829 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862268 4848 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:14.875829 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862271 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:14.875829 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862275 4848 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:14.875829 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862279 4848 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:14.875829 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862283 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:14.875829 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862286 4848 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:14.875829 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862289 4848 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:14.875829 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862292 4848 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:14.875829 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862295 4848 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:14.876304 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862297 4848 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:14.876304 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862300 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:14.876304 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862303 4848 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:14.876304 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862306 4848 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:14.876304 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862308 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:14.876304 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862311 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:14.876304 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862313 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:14.876304 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862316 4848 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:14.876304 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862318 4848 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:14.876304 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862321 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:14.876304 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862324 4848 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:14.876304 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862326 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:14.876304 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862329 4848 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:14.876304 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862331 4848 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:14.876304 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862334 4848 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:14.876304 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862336 4848 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:14.876304 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862339 4848 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:14.876304 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862341 4848 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:14.876304 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862344 4848 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:14.876780 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862346 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:14.876780 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862349 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:14.876780 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862351 4848 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:14.876780 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.862379 4848 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:38:14.876780 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862540 4848 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:14.876780 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862545 4848 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:14.876780 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862548 4848 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:14.876780 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862551 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:14.876780 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862553 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:14.876780 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862557 4848 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:14.876780 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862560 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:14.876780 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862563 4848 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:14.876780 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862565 4848 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:14.876780 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862568 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:14.876780 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862570 4848 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:14.876780 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862573 4848 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:14.877192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862576 4848 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:14.877192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862579 4848 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:14.877192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862582 4848 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:14.877192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862584 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:14.877192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862587 4848 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:14.877192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862590 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:14.877192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862592 4848 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:14.877192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862595 4848 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:14.877192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862597 4848 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:14.877192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862600 4848 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:14.877192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862603 4848 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:14.877192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862606 4848 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:14.877192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862608 4848 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:14.877192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862611 4848 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:14.877192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862613 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:14.877192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862616 4848 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:14.877192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862618 4848 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:14.877192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862621 4848 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:14.877192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862623 4848 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:14.877192 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862626 4848 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:14.877866 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862628 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:14.877866 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862631 4848 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:14.877866 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862633 4848 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:14.877866 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862636 4848 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:14.877866 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862638 4848 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:14.877866 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862641 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:14.877866 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862644 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:14.877866 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862647 4848 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:14.877866 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862650 4848 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:14.877866 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862652 4848 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:14.877866 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862655 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:14.877866 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862658 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:14.877866 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862660 4848 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:14.877866 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862663 4848 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:14.877866 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862665 4848 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:14.877866 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862668 4848 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:14.877866 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862670 4848 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:14.877866 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862673 4848 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:14.877866 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862675 4848 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:14.878658 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862678 4848 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:14.878658 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862680 4848 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:14.878658 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862683 4848 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:14.878658 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862685 4848 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:14.878658 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862688 4848 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:14.878658 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862690 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:14.878658 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862693 4848 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:14.878658 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862695 4848 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:14.878658 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862698 4848 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:14.878658 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862700 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:14.878658 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862703 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:14.878658 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862706 4848 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:14.878658 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862708 4848 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:14.878658 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862711 4848 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:14.878658 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862715 4848 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:14.878658 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862718 4848 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:14.878658 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862721 4848 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:14.878658 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862723 4848 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:14.878658 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862726 4848 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:14.879244 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862729 4848 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:14.879244 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862732 4848 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:14.879244 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862735 4848 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:14.879244 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862737 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:14.879244 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862740 4848 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:14.879244 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862742 4848 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:14.879244 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862745 4848 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:14.879244 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862748 4848 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:14.879244 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862750 4848 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:14.879244 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862753 4848 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:14.879244 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862755 4848 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:14.879244 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862758 4848 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:14.879244 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862760 4848 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:14.879244 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862781 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:14.879244 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862784 4848 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:14.879244 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862787 4848 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:14.879244 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862790 4848 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:14.879244 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862793 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:14.879244 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862796 4848 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:14.879244 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862798 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:14.879728 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862801 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:14.879728 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862803 4848 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:14.879728 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862806 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:14.879728 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862810 4848 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:14.879728 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862814 4848 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:14.879728 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862817 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:14.879728 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862820 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:14.879728 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862823 4848 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:14.879728 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862826 4848 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:14.879728 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862828 4848 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:14.879728 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862831 4848 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:14.879728 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862834 4848 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:14.879728 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862836 4848 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:14.879728 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862839 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:14.879728 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862842 4848 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:14.879728 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862845 4848 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:14.880137 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.862869 4848 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:38:14.880137 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862901 4848 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:38:14.880137 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862905 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:38:14.880137 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862908 4848 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:38:14.880137 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862911 4848 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:38:14.880137 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862914 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:38:14.880137 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862917 4848 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:38:14.880137 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862920 4848 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:38:14.880137 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862922 4848 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:38:14.880137 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862925 4848 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:38:14.880137 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862928 4848 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:38:14.880137 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862931 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:38:14.880137 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862934 4848 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:38:14.880137 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862936 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:38:14.880137 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862939 4848 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:38:14.880137 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862942 4848 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:38:14.880527 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862944 4848 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:38:14.880527 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862947 4848 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:38:14.880527 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862950 4848 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:38:14.880527 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862952 4848 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:38:14.880527 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862955 4848 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:38:14.880527 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862957 4848 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:38:14.880527 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862960 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:38:14.880527 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862962 4848 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:38:14.880527 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862965 4848 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:38:14.880527 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862967 4848 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:38:14.880527 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862970 4848 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:38:14.880527 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862972 4848 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:38:14.880527 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862975 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:38:14.880527 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862977 4848 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:38:14.880527 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862980 4848 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:38:14.880527 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862983 4848 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:38:14.880527 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862986 4848 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:38:14.880527 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862988 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:38:14.880527 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862991 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:38:14.880527 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862993 4848 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:38:14.881038 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862996 4848 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:38:14.881038 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.862998 4848 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:38:14.881038 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863001 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:38:14.881038 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863004 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:38:14.881038 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863006 4848 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:38:14.881038 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863009 4848 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:38:14.881038 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863011 4848 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:38:14.881038 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863018 4848 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:38:14.881038 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863021 4848 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:38:14.881038 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863023 4848 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:38:14.881038 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863026 4848 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:38:14.881038 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863028 4848 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:38:14.881038 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863031 4848 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:38:14.881038 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863034 4848 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:38:14.881038 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863036 4848 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:38:14.881038 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863039 4848 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:38:14.881038 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863041 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:38:14.881038 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863043 4848 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:38:14.881038 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863046 4848 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:38:14.881495 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863049 4848 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:38:14.881495 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863051 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:38:14.881495 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863054 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:38:14.881495 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863056 4848 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:38:14.881495 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863059 4848 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:38:14.881495 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863061 4848 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:38:14.881495 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863065 4848 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:38:14.881495 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863069 4848 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:38:14.881495 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863072 4848 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:38:14.881495 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863075 4848 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:38:14.881495 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863078 4848 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:38:14.881495 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863081 4848 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:38:14.881495 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863084 4848 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:38:14.881495 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863087 4848 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:38:14.881495 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863090 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:38:14.881495 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863093 4848 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:38:14.881495 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863095 4848 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:38:14.881495 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863099 4848 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:38:14.881495 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863102 4848 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:38:14.882010 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863105 4848 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:38:14.882010 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863109 4848 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:38:14.882010 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863112 4848 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:38:14.882010 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863114 4848 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:38:14.882010 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863117 4848 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:38:14.882010 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863120 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:38:14.882010 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863122 4848 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:38:14.882010 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863125 4848 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:38:14.882010 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863128 4848 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:38:14.882010 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863130 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:38:14.882010 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863133 4848 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:38:14.882010 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863136 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:38:14.882010 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863138 4848 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:38:14.882010 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863141 4848 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:38:14.882010 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863144 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:38:14.882010 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863147 4848 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:38:14.882010 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863151 4848 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:38:14.882010 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863154 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:38:14.882010 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863157 4848 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:38:14.882467 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863160 4848 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:38:14.882467 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863164 4848 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:38:14.882467 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863166 4848 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:38:14.882467 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863169 4848 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:38:14.882467 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863173 4848 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:38:14.882467 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863175 4848 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:38:14.882467 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863178 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:38:14.882467 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863180 4848 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:38:14.882467 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863183 4848 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:38:14.882467 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863185 4848 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:38:14.882467 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863188 4848 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:38:14.882467 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863190 4848 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:38:14.882467 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863193 4848 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:38:14.882467 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:14.863195 4848 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:38:14.882467 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.863249 4848 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 20:38:14.882467 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.864261 4848 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 20:38:14.882467 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.864318 4848 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 20:38:14.882467 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.864499 4848 server.go:1014] "Starting client certificate rotation" Oct 30 20:38:14.882467 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.864636 4848 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 20:38:14.882986 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.864685 4848 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:32:29 +0000 UTC" deadline="2028-05-20 22:46:58.536526841 +0000 UTC" Oct 30 20:38:14.882986 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.864712 4848 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13634h8m43.671816312s" Oct 30 20:38:14.882986 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.864959 4848 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:38:14.882986 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.865983 4848 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:38:14.882986 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.867356 4848 log.go:25] "Validated CRI v1 runtime API" Oct 30 20:38:14.882986 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.869566 4848 log.go:25] "Validated CRI v1 image API" Oct 30 20:38:14.882986 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.870285 4848 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 20:38:14.882986 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.871989 4848 fs.go:135] Filesystem UUIDs: map[1d655983-1288-4cda-9453-a48fd6e711c1:/dev/nvme0n1p4 32dcfa16-29c8-4c0f-b136-74d27f27f410:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 20:38:14.883261 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.872011 4848 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/004bdc84b9dfb4f483b8235c30cc031935a4f53a22787b2b8a231cf5b72e1395/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/004bdc84b9dfb4f483b8235c30cc031935a4f53a22787b2b8a231cf5b72e1395/userdata/shm major:0 minor:272 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0eeca3a5e349b85acd986b5c4feba101508cbd8b9205aa426f4b34ae31975955/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0eeca3a5e349b85acd986b5c4feba101508cbd8b9205aa426f4b34ae31975955/userdata/shm major:0 minor:110 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d8faca17500c54dbeeb859698812b94b33e516f72c0b1570ce36af28eba44c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d8faca17500c54dbeeb859698812b94b33e516f72c0b1570ce36af28eba44c2/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31b3722fcbc274c096b600beb5b1892800474f1df99d66e110a6f9b8929fd4a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31b3722fcbc274c096b600beb5b1892800474f1df99d66e110a6f9b8929fd4a0/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d0a8452e963c37b0d11205dc2df943eeba902182d469728a1f8471f255da9e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d0a8452e963c37b0d11205dc2df943eeba902182d469728a1f8471f255da9e9/userdata/shm major:0 minor:274 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4de8b4f4919804deda81fcd89fb21765d52641e297c0176b000d39775f8d57f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4de8b4f4919804deda81fcd89fb21765d52641e297c0176b000d39775f8d57f7/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6699dc1c4ebf7b70abd16bb5b79b34127d1801b261cf3b6e50ad3abbaee3b1db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6699dc1c4ebf7b70abd16bb5b79b34127d1801b261cf3b6e50ad3abbaee3b1db/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66adbecdad87f0fa75c8faa791afe3b11c48828e61636773b808a946920c932e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66adbecdad87f0fa75c8faa791afe3b11c48828e61636773b808a946920c932e/userdata/shm major:0 minor:99 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/755525b3aeedd7c5f9ac6757f8ed83741be4815d1cb49602e62cc30d9344f271/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/755525b3aeedd7c5f9ac6757f8ed83741be4815d1cb49602e62cc30d9344f271/userdata/shm major:0 minor:103 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89858e985189f0e775c5a1b655747c867d52234b23c1f97cf54f3636fbc9e0f7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89858e985189f0e775c5a1b655747c867d52234b23c1f97cf54f3636fbc9e0f7/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4fc79cde41391c8690bb276a888b958b0fb62293a146794e3ec73f90451e3f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4fc79cde41391c8690bb276a888b958b0fb62293a146794e3ec73f90451e3f4/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b7cf93a51c0fd37a6cbae375e98663034a455e030f5815c51c03b85642ed1aae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b7cf93a51c0fd37a6cbae375e98663034a455e030f5815c51c03b85642ed1aae/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c107fdaadcf3cd5d859b5fa6bb4a7c89781203673077a42f0462fc84ee9459f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c107fdaadcf3cd5d859b5fa6bb4a7c89781203673077a42f0462fc84ee9459f4/userdata/shm major:0 minor:270 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5d7feb6f3ddb313ad1539d55cf36b8dc12079ff539dc6538e3bdb87a0144d2b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5d7feb6f3ddb313ad1539d55cf36b8dc12079ff539dc6538e3bdb87a0144d2b/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d722d916e270501ded9f2d355bc57579a0a92010a113ca8ff4f2559a831fab2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d722d916e270501ded9f2d355bc57579a0a92010a113ca8ff4f2559a831fab2c/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7d808b3d42ff570260ca243f48eef40b51f28d66b796b3b743f4fb5b2498ad1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7d808b3d42ff570260ca243f48eef40b51f28d66b796b3b743f4fb5b2498ad1/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ebad2169c8c3c0fbac5433d4c72760bf94ff4a03186a0c4be91d54b618706cef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ebad2169c8c3c0fbac5433d4c72760bf94ff4a03186a0c4be91d54b618706cef/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0137d1fc-8fa3-4b0e-8821-4776913088f7/volumes/kubernetes.io~projected/kube-api-access-rbdm6:{mountpoint:/var/lib/kubelet/pods/0137d1fc-8fa3-4b0e-8821-4776913088f7/volumes/kubernetes.io~projected/kube-api-access-rbdm6 major:0 minor:69 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0832230b-2d6a-40f2-8d5f-dc1da218c7b8/volumes/kubernetes.io~projected/kube-api-access-v7456:{mountpoint:/var/lib/kubelet/pods/0832230b-2d6a-40f2-8d5f-dc1da218c7b8/volumes/kubernetes.io~projected/kube-api-access-v7456 major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/313047e7-bc02-42eb-933b-ab52575af3f7/volumes/kubernetes.io~projected/kube-api-access-9bsb9:{mountpoint:/var/lib/kubelet/pods/313047e7-bc02-42eb-933b-ab52575af3f7/volumes/kubernetes.io~projected/kube-api-access-9bsb9 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c57ecae-89cb-4901-8f9e-e4586771f5d8/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/4c57ecae-89cb-4901-8f9e-e4586771f5d8/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249/volumes/kubernetes.io~projected/kube-api-access-v69k5:{mountpoint:/var/lib/kubelet/pods/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249/volumes/kubernetes.io~projected/kube-api-access-v69k5 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249/volumes/kubernetes.io~secret/metrics-certs major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b91b1fa-0886-4a2c-8005-bd6f77666874/volumes/kubernetes.io~projected/kube-api-access-swthh:{mountpoint:/var/lib/kubelet/pods/5b91b1fa-0886-4a2c-8005-bd6f77666874/volumes/kubernetes.io~projected/kube-api-access-swthh major:0 minor:68 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ce76c71-8b23-49b4-a61c-9b14adea86c7/volumes/kubernetes.io~projected/kube-api-access-5mcpr:{mountpoint:/var/lib/kubelet/pods/6ce76c71-8b23-49b4-a61c-9b14adea86c7/volumes/kubernetes.io~projected/kube-api-access-5mcpr major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9224f6c0-bc3e-4eee-a78b-431e6c8f4705/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/9224f6c0-bc3e-4eee-a78b-431e6c8f4705/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9224f6c0-bc3e-4eee-a78b-431e6c8f4705/volumes/kubernetes.io~projected/kube-api-access-7662k:{mountpoint:/var/lib/kubelet/pods/9224f6c0-bc3e-4eee-a78b-431e6c8f4705/volumes/kubernetes.io~projected/kube-api-access-7662k major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9224f6c0-bc3e-4eee-a78b-431e6c8f4705/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/9224f6c0-bc3e-4eee-a78b-431e6c8f4705/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9390b293-dd28-4713-8452-b9fb5d7ccaec/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/9390b293-dd28-4713-8452-b9fb5d7ccaec/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9390b293-dd28-4713-8452-b9fb5d7ccaec/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/9390b293-dd28-4713-8452-b9fb5d7ccaec/volumes/kubernetes.io~empty-dir/tmp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9390b293-dd28-4713-8452-b9fb5d7ccaec/volumes/kubernetes.io~projected/kube-api-access-lkqcq:{mountpoint:/var/lib/kubelet/pods/9390b293-dd28-4713-8452-b9fb5d7ccaec/volumes/kubernetes.io~projected/kube-api-access-lkqcq major:0 minor:70 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8e2605f-358c-4fd1-ae8a-524282c76f72/volumes/kubernetes.io~projected/kube-api-access-s7ns5:{mountpoint:/var/lib/kubelet/pods/a8e2605f-358c-4fd1-ae8a-524282c76f72/volumes/kubernetes.io~projected/kube-api-access-s7ns5 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8e2605f-358c-4fd1-ae8a-524282c76f72/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a8e2605f-358c-4fd1-ae8a-524282c76f72/volumes/kubernetes.io~secret/cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a97f29d4-5d15-432a-a493-6011cf8e059e/volumes/kubernetes.io~projected/kube-api-access-dm59l:{mountpoint:/var/lib/kubelet/pods/a97f29d4-5d15-432a-a493-6011cf8e059e/volumes/kubernetes.io~projected/kube-api-access-dm59l major:0 minor:109 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a97f29d4-5d15-432a-a493-6011cf8e059e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a97f29d4-5d15-432a-a493-6011cf8e059e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:108 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a97f29d4-5d15-432a-a493-6011cf8e059e/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/a97f29d4-5d15-432a-a493-6011cf8e059e/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:107 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a99706d1-d012-4dd9-b46a-5ec139287351/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a99706d1-d012-4dd9-b46a-5ec139287351/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab625ab1-de4f-4418-bd65-edb628d963fe/volumes/kubernetes.io~projected/kube-api-access-7krsp:{mountpoint:/var/lib/kubelet/pods/ab625ab1-de4f-4418-bd65-edb628d963fe/volumes/kubernetes.io~projected/kube-api-access-7krsp major:0 minor:71 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b66dabb7-8606-4898-99ca-954ffd602da2/volumes/kubernetes.io~projected/kube-api-access-mqs5f:{mountpoint:/var/lib/kubelet/pods/b66dabb7-8606-4898-99ca-954ffd602da2/volumes/kubernetes.io~projected/kube-api-access-mqs5f major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b66dabb7-8606-4898-99ca-954ffd602da2/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/b66dabb7-8606-4898-99ca-954ffd602da2/volumes/kubernetes.io~secret/metrics-tls major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da9ceed2-3ab8-43dc-88a3-f20d7351ac30/volumes/kubernetes.io~projected/kube-api-access-j47v7:{mountpoint:/var/lib/kubelet/pods/da9ceed2-3ab8-43dc-88a3-f20d7351ac30/volumes/kubernetes.io~projected/kube-api-access-j47v7 major:0 minor:72 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbe199fd-1f79-4fa3-9136-21a58fa1202d/volumes/kubernetes.io~projected/kube-api-access-slwwv:{mountpoint:/var/lib/kubelet/pods/dbe199fd-1f79-4fa3-9136-21a58fa1202d/volumes/kubernetes.io~projected/kube-api-access-slwwv major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbe199fd-1f79-4fa3-9136-21a58fa1202d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/dbe199fd-1f79-4fa3-9136-21a58fa1202d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:218 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-101:{mountpoint:/var/lib/containers/storage/overlay/450cb7b481a14c378a10995788fad5772a9bddda1382f2941c1922a5c5493953/merged major:0 minor:101 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/d8dad75beb42a2e3e8d07a9fa5b772aa635718b26225d2be5c2619861513545f/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-112:{mountpoint:/var/lib/containers/storage/overlay/7f017b6c2717fa9a4e92f353885d4d64e486dd2a05b808f00b12f182e5e1e839/merged major:0 minor:112 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/c9492f9bfd03fe7adc416256b106a4e61a21a21e8a77280011f4e98ef6a58af5/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-116:{mountpoint:/var/lib/containers/storage/overlay/459d2e96eb66d0b6edd94f7a30e7b6f4db1ab86c1aa112d888cff9c5f29b90ee/merged major:0 minor:116 fsType:overlay blockSize:0} overlay_0-121:{mountpoint:/var/lib/containers/storage/overlay/47c53b49421561f294ee2f7f6a0bcf063c9aeb61cbcdf387fad64fd925dcfcce/merged major:0 minor:121 fsType:overlay blockSize:0} overlay_0-123:{mountpoint:/var/lib/containers/storage/overlay/24eaa315f35d2485a3c53b2bc1c346058a992079bf8d988ad597be3493828139/merged major:0 minor:123 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/2d6719d79e2bf15e3b425a56cb7f83115b8245088c78bdb695f2c9f2b8368a5c/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/95974faf45d3b6e8066aefa5778cce4cedcab8336ee0ea5e65a1616fc370fed7/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/0f0e11d89b6b3849483ae8c114f332625a515f6b849a48bb2b6e2dadaf452c1b/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/dbf86a236da55014120099ce84a91ad82437502972716a5c92e7c5a2785e07de/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/0882503835fda92fa7a1abe36be577eeab8422e0aea39b586ca1e57a5d5d7a4c/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-150:{mountpoint:/var/lib/containers/storage/overlay/db9e96d41838fbfa7b8109e56209cb95a3c09c88ff1f4cabd6a07e6e0a9257af/merged major:0 minor:150 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/327aebf11b62c72f5ecd4b90bc1d301cd38b1a0d7992abff663f4ce7b1046b04/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/cc44c780f3c26591947f6213ae09ecde1cc067db5d8904a0e480df59db94fe43/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-173:{mountpoint:/var/lib/containers/storage/overlay/8371100d400157fcc6724a3b9f774a1ff218ef9bec19ededb23ee76168612e94/merged major:0 minor:173 fsType:overlay blockSize:0} overlay_0-178:{mountpoint:/var/lib/containers/storage/overlay/c9af498395695871ee6955a6772c031a72470bae802bc8e5722c9a6b084e2464/merged major:0 minor:178 fsType:overlay blockSize:0} overlay_0-183:{mountpoint:/var/lib/containers/storage/overlay/b858db48962adaf932ac85577af3fb098f7898fd2e28ba37f336bab40251883c/merged major:0 minor:183 fsType:overlay blockSize:0} overlay_0-188:{mountpoint:/var/lib/containers/storage/overlay/f3f50158ed08e5bc61c411e9eef50ffefff690efd5f117696c6923cb2ea161bd/merged major:0 minor:188 fsType:overlay blockSize:0} overlay_0-193:{mountpoint:/var/lib/containers/storage/overlay/ca3460711385c045570a9d78ed22ae5711eaa5e45f71d6a8a121e053f59dba07/merged major:0 minor:193 fsType:overlay blockSize:0} overlay_0-198:{mountpoint:/var/lib/containers/storage/overlay/afd5d70856b4213cdcd59f43c2ee59e041110600612185762c34be5030b0fb68/merged major:0 minor:198 fsType:overlay blockSize:0} overlay_0-203:{mountpoint:/var/lib/containers/storage/overlay/dfbaed802cfa7ce98854e7ffe9e90e2d887347c5feafc1ff0ce1ab0da7fa9a4f/merged major:0 minor:203 fsType:overlay blockSize:0} overlay_0-208:{mountpoint:/var/lib/containers/storage/overlay/9a756e9933334e4f56fe6b47a60a06e3c12f55d36353506f724e21f74d84bbe2/merged major:0 minor:208 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/fa6b67bec31acd6111329891648d4285c5a6cbbcad3b76eb3d7cb9430500b7d4/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/d986ad0381de20b29e1bcbf7fb657fbb197dd31ca33e3138535c94a5a0d11af6/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/ca056f28ac630b6bda7add4203cb344864acb8ce141193a4c0d01a908603a48d/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/6fb95833dd2fb3dc6f91ae40f37a738a652a938474136bacf1ea276ec2d9e68a/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/e2b22b4091f9e4ceeaa1caa528a82a996e1cc93133560bfb5f1bd3abce2b6fa7/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/9012a322e8e30e72db2a942fdd5fa3ca7178d710ab8a7329365b29e204f5b000/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/84cfc7d2a911eaff96c23ff59c1a6b5f5e225b1834dcacac2a2f019c9a9ace36/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/c0f0a15009e230d78912d10e70c2cec3ba0398095090d3bdb832c135cd4d8c4a/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/a877597499e94a7e9035d5f10a4c5a8f3ace976baf3e2192818a71a7265f326e/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/ac8dac7508e4df490d181ae291d71201da4f1c095f8527d04ed7c97584e7dcaf/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/2daf198283d8dac67625868d4dc35f9d4be762e6c7293f4a224af6bab8eb38b7/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/e2bd12009a81aed0772d138207096b8504bc2146a3e0c4e5e54d6c7ba8740d06/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/70cb6796eae0f9059eea97b28f1145d9121d03b6f7ae066965c72f8ef1aa21aa/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/81adf39a56c5b9362fda500665e8fa92ed884fee5ad22c33652e68f62dfed366/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/dd249301184361c814d11ba4c4eb6cc1243dd017a318d5a2e202ddba9c724c28/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/0d81ff4e858090631903b306d19db832f39d6d4699ded70ae67c060c9a6d4b9d/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/7cae82ca15383b65484d2ce1468ca94be1e97a43d44e061ac7232852c5148f3f/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/79e2290b47268910cf8eba35b6c135625179499296e2041806a17d5f406026ca/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/bfb1826fd03db7414cb303b836224717a1519032b91d9de4ac2b958b99022c67/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-57:{mountpoint:/var/lib/containers/storage/overlay/a3b7191276bd925a0aeb8a848c2852e7601d54fa1865e2966959bbf72ed18789/merged major:0 minor:57 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/f5a0bd1868dbf4b32af2c2217b5387ae8ccea22c1f5a437ccedf1749f731c640/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/150d19a648ac2af80dc7f7a5e3835edb167591bd208327c46e6f0b6da53b9b5c/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/5d550648f39d1fea6ccf990419bca0e45ef245824abb47208e06633f2ec61c61/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-87:{mountpoint:/var/lib/containers/storage/overlay/09be25617fdd75224335f6867c9aa6420636a6d7b16b72f9362b2e4705b7b743/merged major:0 minor:87 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/c32b846db5dcdf248342e984af4420a42d77d0bc5a323293ef2d13b4912d3993/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-95:{mountpoint:/var/lib/containers/storage/overlay/350afee1a35a68a18575a9631b5d224a1d95e69df680cbdc6de7b45223ef1279/merged major:0 minor:95 fsType:overlay blockSize:0} overlay_0-97:{mountpoint:/var/lib/containers/storage/overlay/15fbb2fdc8a3f8459eccaa5a8649ef321f5babb0d96755d1924f310d6e3e1a84/merged major:0 minor:97 fsType:overlay blockSize:0}] Oct 30 20:38:14.883435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.880468 4848 manager.go:217] Machine: {Timestamp:2026-10-30 20:38:14.880006297 +0000 UTC m=+0.059902795 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100049 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26dece88547659e225b6460b365b83 SystemUUID:ec26dece-8854-7659-e225-b6460b365b83 BootID:c4f62b97-d062-418a-a4a7-44b6f49b9276 Filesystems:[{Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4c57ecae-89cb-4901-8f9e-e4586771f5d8/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4de8b4f4919804deda81fcd89fb21765d52641e297c0176b000d39775f8d57f7/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-203 DeviceMajor:0 DeviceMinor:203 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9224f6c0-bc3e-4eee-a78b-431e6c8f4705/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/a99706d1-d012-4dd9-b46a-5ec139287351/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:258 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b91b1fa-0886-4a2c-8005-bd6f77666874/volumes/kubernetes.io~projected/kube-api-access-swthh DeviceMajor:0 DeviceMinor:68 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dbe199fd-1f79-4fa3-9136-21a58fa1202d/volumes/kubernetes.io~projected/kube-api-access-slwwv DeviceMajor:0 DeviceMinor:220 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-97 DeviceMajor:0 DeviceMinor:97 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0eeca3a5e349b85acd986b5c4feba101508cbd8b9205aa426f4b34ae31975955/userdata/shm DeviceMajor:0 DeviceMinor:110 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-150 DeviceMajor:0 DeviceMinor:150 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-198 DeviceMajor:0 DeviceMinor:198 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8e2605f-358c-4fd1-ae8a-524282c76f72/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ab625ab1-de4f-4418-bd65-edb628d963fe/volumes/kubernetes.io~projected/kube-api-access-7krsp DeviceMajor:0 DeviceMinor:71 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66adbecdad87f0fa75c8faa791afe3b11c48828e61636773b808a946920c932e/userdata/shm DeviceMajor:0 DeviceMinor:99 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31b3722fcbc274c096b600beb5b1892800474f1df99d66e110a6f9b8929fd4a0/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ce76c71-8b23-49b4-a61c-9b14adea86c7/volumes/kubernetes.io~projected/kube-api-access-5mcpr DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249/volumes/kubernetes.io~projected/kube-api-access-v69k5 DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-95 DeviceMajor:0 DeviceMinor:95 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-193 DeviceMajor:0 DeviceMinor:193 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4fc79cde41391c8690bb276a888b958b0fb62293a146794e3ec73f90451e3f4/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-87 DeviceMajor:0 DeviceMinor:87 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a97f29d4-5d15-432a-a493-6011cf8e059e/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:107 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d722d916e270501ded9f2d355bc57579a0a92010a113ca8ff4f2559a831fab2c/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d8faca17500c54dbeeb859698812b94b33e516f72c0b1570ce36af28eba44c2/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9224f6c0-bc3e-4eee-a78b-431e6c8f4705/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-178 DeviceMajor:0 DeviceMinor:178 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0137d1fc-8fa3-4b0e-8821-4776913088f7/volumes/kubernetes.io~projected/kube-api-access-rbdm6 DeviceMajor:0 DeviceMinor:69 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da9ceed2-3ab8-43dc-88a3-f20d7351ac30/volumes/kubernetes.io~projected/kube-api-access-j47v7 DeviceMajor:0 DeviceMinor:72 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89858e985189f0e775c5a1b655747c867d52234b23c1f97cf54f3636fbc9e0f7/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-123 DeviceMajor:0 DeviceMinor:123 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b66dabb7-8606-4898-99ca-954ffd602da2/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:261 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ebad2169c8c3c0fbac5433d4c72760bf94ff4a03186a0c4be91d54b618706cef/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9224f6c0-bc3e-4eee-a78b-431e6c8f4705/volumes/kubernetes.io~projected/kube-api-access-7662k DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9390b293-dd28-4713-8452-b9fb5d7ccaec/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5d7feb6f3ddb313ad1539d55cf36b8dc12079ff539dc6538e3bdb87a0144d2b/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-101 DeviceMajor:0 DeviceMinor:101 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c107fdaadcf3cd5d859b5fa6bb4a7c89781203673077a42f0462fc84ee9459f4/userdata/shm DeviceMajor:0 DeviceMinor:270 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-57 DeviceMajor:0 DeviceMinor:57 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-173 DeviceMajor:0 DeviceMinor:173 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dbe199fd-1f79-4fa3-9136-21a58fa1202d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b66dabb7-8606-4898-99ca-954ffd602da2/volumes/kubernetes.io~projected/kube-api-access-mqs5f DeviceMajor:0 DeviceMinor:221 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b7cf93a51c0fd37a6cbae375e98663034a455e030f5815c51c03b85642ed1aae/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/313047e7-bc02-42eb-933b-ab52575af3f7/volumes/kubernetes.io~projected/kube-api-access-9bsb9 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a97f29d4-5d15-432a-a493-6011cf8e059e/volumes/kubernetes.io~projected/kube-api-access-dm59l DeviceMajor:0 DeviceMinor:109 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9390b293-dd28-4713-8452-b9fb5d7ccaec/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/755525b3aeedd7c5f9ac6757f8ed83741be4815d1cb49602e62cc30d9344f271/userdata/shm DeviceMajor:0 DeviceMinor:103 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a97f29d4-5d15-432a-a493-6011cf8e059e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:108 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-116 DeviceMajor:0 DeviceMinor:116 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-183 DeviceMajor:0 DeviceMinor:183 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-208 DeviceMajor:0 DeviceMinor:208 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0832230b-2d6a-40f2-8d5f-dc1da218c7b8/volumes/kubernetes.io~projected/kube-api-access-v7456 DeviceMajor:0 DeviceMinor:269 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6699dc1c4ebf7b70abd16bb5b79b34127d1801b261cf3b6e50ad3abbaee3b1db/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/004bdc84b9dfb4f483b8235c30cc031935a4f53a22787b2b8a231cf5b72e1395/userdata/shm DeviceMajor:0 DeviceMinor:272 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d0a8452e963c37b0d11205dc2df943eeba902182d469728a1f8471f255da9e9/userdata/shm DeviceMajor:0 DeviceMinor:274 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9390b293-dd28-4713-8452-b9fb5d7ccaec/volumes/kubernetes.io~projected/kube-api-access-lkqcq DeviceMajor:0 DeviceMinor:70 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7d808b3d42ff570260ca243f48eef40b51f28d66b796b3b743f4fb5b2498ad1/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-112 DeviceMajor:0 DeviceMinor:112 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-121 DeviceMajor:0 DeviceMinor:121 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-188 DeviceMajor:0 DeviceMinor:188 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8e2605f-358c-4fd1-ae8a-524282c76f72/volumes/kubernetes.io~projected/kube-api-access-s7ns5 DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:004bdc84b9dfb4f MacAddress:6e:1e:85:8a:b2:ee Speed:10000 Mtu:8901} {Name:2d8faca17500c54 MacAddress:b6:d9:d8:40:4a:16 Speed:10000 Mtu:8901} {Name:3d0a8452e963c37 MacAddress:8a:4a:6e:31:52:9c Speed:10000 Mtu:8901} {Name:b7cf93a51c0fd37 MacAddress:ea:83:81:74:14:c1 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c1:cd:46:81 Speed:0 Mtu:9001} {Name:br-int MacAddress:3e:23:48:a1:89:84 Speed:0 Mtu:8901} {Name:c107fdaadcf3cd5 MacAddress:e2:14:47:80:d7:ce Speed:10000 Mtu:8901} {Name:d722d916e270501 MacAddress:c6:a0:13:f7:d4:1f Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c1:cd:46:81 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:16:3c:92:68:3e:52 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1e:74:d1:5b:1a:99 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 20:38:14.883435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.880748 4848 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 20:38:14.883435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.880806 4848 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 20:38:14.883435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.881118 4848 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 20:38:14.883435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.881143 4848 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-52.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 20:38:14.883435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.881270 4848 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 20:38:14.883435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.881277 4848 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 20:38:14.883435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.881290 4848 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:38:14.883435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.881307 4848 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:38:14.883435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.881395 4848 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 20:38:14.883435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.881581 4848 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 20:38:14.883435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.881698 4848 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 20:38:14.883435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.881739 4848 kubelet.go:493] "Attempting to sync node with API server" Oct 30 20:38:14.883435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.881755 4848 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 20:38:14.883435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.881780 4848 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 20:38:14.883435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.881793 4848 kubelet.go:405] "Adding apiserver pod source" Oct 30 20:38:14.883435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.881804 4848 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 20:38:14.883435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.882654 4848 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 20:38:14.883435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.882665 4848 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:38:14.883435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.883140 4848 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 20:38:14.885178 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.885165 4848 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 20:38:14.885178 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.885178 4848 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:38:14.885286 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.885277 4848 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 20:38:14.885483 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.885473 4848 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 20:38:14.885513 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.885502 4848 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 20:38:14.885592 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.885583 4848 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 20:38:14.885626 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.885598 4848 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 20:38:14.885626 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.885605 4848 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 20:38:14.885626 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.885612 4848 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 20:38:14.885626 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.885619 4848 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 20:38:14.885721 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.885629 4848 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 20:38:14.885721 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.885638 4848 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 20:38:14.885721 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.885648 4848 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 20:38:14.885721 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.885657 4848 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 20:38:14.885721 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.885666 4848 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 20:38:14.885721 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.885693 4848 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 20:38:14.885721 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.885710 4848 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 20:38:14.885917 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.885736 4848 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 20:38:14.885917 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.885753 4848 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 20:38:14.889014 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.889003 4848 server.go:1263] "Started kubelet" Oct 30 20:38:14.889444 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.889101 4848 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 20:38:14.889444 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.889225 4848 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 20:38:14.889603 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.889542 4848 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 20:38:14.889582 ip-10-0-130-52 systemd[1]: Started Kubernetes Kubelet. Oct 30 20:38:14.889938 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.889908 4848 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 20:38:14.891089 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.891071 4848 server.go:317] "Adding debug handlers to kubelet server" Oct 30 20:38:14.894028 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.893959 4848 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 20:38:14.894515 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.894463 4848 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 20:38:14.894515 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.894486 4848 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 20:38:14.895086 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:38:14.894954 4848 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-52.us-east-2.compute.internal\" not found" Oct 30 20:38:14.895086 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.894466 4848 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 20:38:14.895086 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.895032 4848 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 20:38:14.895559 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.895504 4848 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:32:37 +0000 UTC" deadline="2028-07-17 01:17:10.754281306 +0000 UTC" Oct 30 20:38:14.895715 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.895664 4848 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15004h38m55.858623222s" Oct 30 20:38:14.898962 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.898477 4848 factory.go:55] Registering systemd factory Oct 30 20:38:14.898962 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.898499 4848 factory.go:223] Registration of the systemd container factory successfully Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899307 4848 factory.go:153] Registering CRI-O factory Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899327 4848 factory.go:223] Registration of the crio container factory successfully Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899394 4848 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899416 4848 factory.go:103] Registering Raw factory Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899432 4848 manager.go:1199] Started watching for new ooms in manager Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899544 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9224f6c0-bc3e-4eee-a78b-431e6c8f4705" volumeName="kubernetes.io/configmap/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-env-overrides" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899596 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b66dabb7-8606-4898-99ca-954ffd602da2" volumeName="kubernetes.io/secret/b66dabb7-8606-4898-99ca-954ffd602da2-metrics-tls" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899612 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbe199fd-1f79-4fa3-9136-21a58fa1202d" volumeName="kubernetes.io/projected/dbe199fd-1f79-4fa3-9136-21a58fa1202d-kube-api-access-slwwv" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899630 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0137d1fc-8fa3-4b0e-8821-4776913088f7" volumeName="kubernetes.io/configmap/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-daemon-config" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899641 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="313047e7-bc02-42eb-933b-ab52575af3f7" volumeName="kubernetes.io/projected/313047e7-bc02-42eb-933b-ab52575af3f7-kube-api-access-9bsb9" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899654 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9224f6c0-bc3e-4eee-a78b-431e6c8f4705" volumeName="kubernetes.io/configmap/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-ovnkube-config" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899665 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9390b293-dd28-4713-8452-b9fb5d7ccaec" volumeName="kubernetes.io/projected/9390b293-dd28-4713-8452-b9fb5d7ccaec-kube-api-access-lkqcq" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899687 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da9ceed2-3ab8-43dc-88a3-f20d7351ac30" volumeName="kubernetes.io/projected/da9ceed2-3ab8-43dc-88a3-f20d7351ac30-kube-api-access-j47v7" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899702 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbe199fd-1f79-4fa3-9136-21a58fa1202d" volumeName="kubernetes.io/empty-dir/dbe199fd-1f79-4fa3-9136-21a58fa1202d-data-volume" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899716 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a97f29d4-5d15-432a-a493-6011cf8e059e" volumeName="kubernetes.io/configmap/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899729 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a97f29d4-5d15-432a-a493-6011cf8e059e" volumeName="kubernetes.io/empty-dir/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-textfile" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899743 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0137d1fc-8fa3-4b0e-8821-4776913088f7" volumeName="kubernetes.io/projected/0137d1fc-8fa3-4b0e-8821-4776913088f7-kube-api-access-rbdm6" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899753 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9224f6c0-bc3e-4eee-a78b-431e6c8f4705" volumeName="kubernetes.io/configmap/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-ovnkube-script-lib" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899785 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9224f6c0-bc3e-4eee-a78b-431e6c8f4705" volumeName="kubernetes.io/secret/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899799 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8e2605f-358c-4fd1-ae8a-524282c76f72" volumeName="kubernetes.io/projected/a8e2605f-358c-4fd1-ae8a-524282c76f72-kube-api-access-s7ns5" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899812 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b66dabb7-8606-4898-99ca-954ffd602da2" volumeName="kubernetes.io/empty-dir/b66dabb7-8606-4898-99ca-954ffd602da2-tmp-dir" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899823 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b66dabb7-8606-4898-99ca-954ffd602da2" volumeName="kubernetes.io/projected/b66dabb7-8606-4898-99ca-954ffd602da2-kube-api-access-mqs5f" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899842 4848 manager.go:319] Starting recovery of all containers Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899849 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0832230b-2d6a-40f2-8d5f-dc1da218c7b8" volumeName="kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899863 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9224f6c0-bc3e-4eee-a78b-431e6c8f4705" volumeName="kubernetes.io/projected/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-kube-api-access-7662k" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899875 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a97f29d4-5d15-432a-a493-6011cf8e059e" volumeName="kubernetes.io/secret/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899886 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab625ab1-de4f-4418-bd65-edb628d963fe" volumeName="kubernetes.io/empty-dir/ab625ab1-de4f-4418-bd65-edb628d963fe-tmp-dir" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899897 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbe199fd-1f79-4fa3-9136-21a58fa1202d" volumeName="kubernetes.io/configmap/dbe199fd-1f79-4fa3-9136-21a58fa1202d-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899918 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbe199fd-1f79-4fa3-9136-21a58fa1202d" volumeName="kubernetes.io/secret/dbe199fd-1f79-4fa3-9136-21a58fa1202d-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899931 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b91b1fa-0886-4a2c-8005-bd6f77666874" volumeName="kubernetes.io/configmap/5b91b1fa-0886-4a2c-8005-bd6f77666874-iptables-alerter-script" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899942 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ce76c71-8b23-49b4-a61c-9b14adea86c7" volumeName="kubernetes.io/configmap/6ce76c71-8b23-49b4-a61c-9b14adea86c7-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899954 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9390b293-dd28-4713-8452-b9fb5d7ccaec" volumeName="kubernetes.io/empty-dir/9390b293-dd28-4713-8452-b9fb5d7ccaec-tmp" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899965 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b66dabb7-8606-4898-99ca-954ffd602da2" volumeName="kubernetes.io/configmap/b66dabb7-8606-4898-99ca-954ffd602da2-config-volume" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899976 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da9ceed2-3ab8-43dc-88a3-f20d7351ac30" volumeName="kubernetes.io/configmap/da9ceed2-3ab8-43dc-88a3-f20d7351ac30-serviceca" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.899992 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b91b1fa-0886-4a2c-8005-bd6f77666874" volumeName="kubernetes.io/projected/5b91b1fa-0886-4a2c-8005-bd6f77666874-kube-api-access-swthh" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.900065 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8e2605f-358c-4fd1-ae8a-524282c76f72" volumeName="kubernetes.io/secret/a8e2605f-358c-4fd1-ae8a-524282c76f72-cert" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.900120 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a97f29d4-5d15-432a-a493-6011cf8e059e" volumeName="kubernetes.io/configmap/a97f29d4-5d15-432a-a493-6011cf8e059e-metrics-client-ca" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.900136 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249" volumeName="kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.900148 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ce76c71-8b23-49b4-a61c-9b14adea86c7" volumeName="kubernetes.io/configmap/6ce76c71-8b23-49b4-a61c-9b14adea86c7-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.900167 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ce76c71-8b23-49b4-a61c-9b14adea86c7" volumeName="kubernetes.io/configmap/6ce76c71-8b23-49b4-a61c-9b14adea86c7-cni-binary-copy" seLinuxMountContext="" Oct 30 20:38:14.900158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.900179 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ce76c71-8b23-49b4-a61c-9b14adea86c7" volumeName="kubernetes.io/projected/6ce76c71-8b23-49b4-a61c-9b14adea86c7-kube-api-access-5mcpr" seLinuxMountContext="" Oct 30 20:38:14.904919 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.900193 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a99706d1-d012-4dd9-b46a-5ec139287351" volumeName="kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret" seLinuxMountContext="" Oct 30 20:38:14.904919 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.900205 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0137d1fc-8fa3-4b0e-8821-4776913088f7" volumeName="kubernetes.io/configmap/0137d1fc-8fa3-4b0e-8821-4776913088f7-cni-binary-copy" seLinuxMountContext="" Oct 30 20:38:14.904919 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.900217 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c57ecae-89cb-4901-8f9e-e4586771f5d8" volumeName="kubernetes.io/secret/4c57ecae-89cb-4901-8f9e-e4586771f5d8-agent-certs" seLinuxMountContext="" Oct 30 20:38:14.904919 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.900228 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9390b293-dd28-4713-8452-b9fb5d7ccaec" volumeName="kubernetes.io/empty-dir/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-tuned" seLinuxMountContext="" Oct 30 20:38:14.904919 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.900239 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a97f29d4-5d15-432a-a493-6011cf8e059e" volumeName="kubernetes.io/projected/a97f29d4-5d15-432a-a493-6011cf8e059e-kube-api-access-dm59l" seLinuxMountContext="" Oct 30 20:38:14.904919 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.900258 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a97f29d4-5d15-432a-a493-6011cf8e059e" volumeName="kubernetes.io/secret/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-tls" seLinuxMountContext="" Oct 30 20:38:14.904919 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.900289 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab625ab1-de4f-4418-bd65-edb628d963fe" volumeName="kubernetes.io/projected/ab625ab1-de4f-4418-bd65-edb628d963fe-kube-api-access-7krsp" seLinuxMountContext="" Oct 30 20:38:14.904919 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.900301 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c57ecae-89cb-4901-8f9e-e4586771f5d8" volumeName="kubernetes.io/configmap/4c57ecae-89cb-4901-8f9e-e4586771f5d8-konnectivity-ca" seLinuxMountContext="" Oct 30 20:38:14.904919 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.900313 4848 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249" volumeName="kubernetes.io/projected/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-kube-api-access-v69k5" seLinuxMountContext="" Oct 30 20:38:14.904919 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.900326 4848 reconstruct.go:97] "Volume reconstruction finished" Oct 30 20:38:14.904919 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.900334 4848 reconciler.go:29] "Reconciler: start to sync state" Oct 30 20:38:14.910416 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.910293 4848 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:38:14.910984 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.910961 4848 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:38:14.911186 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.911167 4848 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 20:38:14.913283 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.913261 4848 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:38:14.922916 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.922875 4848 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 20:38:14.924165 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.924141 4848 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 20:38:14.924269 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.924172 4848 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 20:38:14.924269 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.924194 4848 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 20:38:14.924269 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:38:14.924248 4848 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 20:38:14.926886 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.926862 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 20:38:14.928894 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.928873 4848 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:38:14.929004 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.928983 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 20:38:14.944304 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.944272 4848 generic.go:356] "Generic (PLEG): container finished" podID="0459ad6c6db5719be39646515856f6a3" containerID="86c55d4aea6a923581be42279421c6a7d7627e21c4599cc522d38f8b2bee8cda" exitCode=0 Oct 30 20:38:14.952340 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.952317 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 20:38:14.952652 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.952631 4848 generic.go:356] "Generic (PLEG): container finished" podID="313047e7-bc02-42eb-933b-ab52575af3f7" containerID="28a7be43fdd06975ec9032da3278f730afa1ea299f5b5d56ae2562daf3a98094" exitCode=1 Oct 30 20:38:14.963257 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.963221 4848 generic.go:356] "Generic (PLEG): container finished" podID="a97f29d4-5d15-432a-a493-6011cf8e059e" containerID="f90eb0ea414935aebd2e8458d6fce14a09e5f821d6732cf3d219dfd3b1bfb2f6" exitCode=0 Oct 30 20:38:14.966938 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.966906 4848 generic.go:356] "Generic (PLEG): container finished" podID="6ce76c71-8b23-49b4-a61c-9b14adea86c7" containerID="feed76ffb0a57996a6e070056d4f73660fcc8f2bf1b83847546edd0bba42dd54" exitCode=0 Oct 30 20:38:14.966938 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.966926 4848 generic.go:356] "Generic (PLEG): container finished" podID="6ce76c71-8b23-49b4-a61c-9b14adea86c7" containerID="966df842b85647b2e8a8ff0d0ecafe62c24585be0de4c1d0712f02641b51c2b0" exitCode=0 Oct 30 20:38:14.966938 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.966932 4848 generic.go:356] "Generic (PLEG): container finished" podID="6ce76c71-8b23-49b4-a61c-9b14adea86c7" containerID="563d32af2643c0ad4dd76de6c65f1fc64ca8e36155d3c99127cb3b944b63fcf6" exitCode=0 Oct 30 20:38:14.966938 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.966938 4848 generic.go:356] "Generic (PLEG): container finished" podID="6ce76c71-8b23-49b4-a61c-9b14adea86c7" containerID="1bc2a30768271765c99d111719a0ef275ccab7276fadd54368598bf12845129a" exitCode=0 Oct 30 20:38:14.967214 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.966946 4848 generic.go:356] "Generic (PLEG): container finished" podID="6ce76c71-8b23-49b4-a61c-9b14adea86c7" containerID="e462dd2bc35cb1bbbe0a3a8f39872f60eefa95e5d915139ff2b7ac16d78a8a0e" exitCode=0 Oct 30 20:38:14.967214 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.966954 4848 generic.go:356] "Generic (PLEG): container finished" podID="6ce76c71-8b23-49b4-a61c-9b14adea86c7" containerID="da08cdc8b0c8d5eeddea290c41ae4d31fb25ab9fb6dd3353abfdd7af00dfd5b9" exitCode=0 Oct 30 20:38:14.978586 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.978564 4848 manager.go:324] Recovery completed Oct 30 20:38:14.996206 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.996183 4848 cpu_manager.go:226] "Starting" policy="none" Oct 30 20:38:14.996206 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.996202 4848 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 20:38:14.996388 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.996222 4848 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 20:38:14.996388 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.996381 4848 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 20:38:14.996502 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.996397 4848 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 20:38:14.996502 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.996422 4848 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 20:38:14.996502 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.996431 4848 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 20:38:14.996502 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.996439 4848 policy_none.go:50] "Start" Oct 30 20:38:14.996502 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.996451 4848 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 20:38:14.996502 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.996464 4848 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 20:38:14.996860 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.996640 4848 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 20:38:14.996860 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.996651 4848 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 20:38:14.996860 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:14.996659 4848 policy_none.go:44] "Start" Oct 30 20:38:15.000757 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.000728 4848 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 20:38:15.004093 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.004080 4848 manager.go:341] "Starting Device Plugin manager" Oct 30 20:38:15.004752 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:38:15.004103 4848 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 20:38:15.004822 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.004757 4848 server.go:87] "Starting device plugin registration server" Oct 30 20:38:15.005116 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.005104 4848 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 20:38:15.005147 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.005119 4848 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 20:38:15.005216 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.005196 4848 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 20:38:15.005334 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.005298 4848 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 20:38:15.005395 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.005345 4848 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 20:38:15.005395 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.005352 4848 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 20:38:15.005481 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.005398 4848 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:38:15.005332349Z","UUID":"05a10d44-1805-4198-a710-98ada9b64a9a","Handler":null,"Name":"","Endpoint":""} Oct 30 20:38:15.006651 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:38:15.006629 4848 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 20:38:15.007570 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.007550 4848 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 20:38:15.007570 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.007574 4848 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 20:38:15.024397 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.024307 4848 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal"] Oct 30 20:38:15.024989 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.024975 4848 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:15.025039 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.025002 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" event={"ID":"0459ad6c6db5719be39646515856f6a3","Type":"ContainerStarted","Data":"885f87af00e1312a5de4e66c046c8dcd02f79954e5a2083be4e8cc7c1aa77a67"} Oct 30 20:38:15.025078 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.025054 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" event={"ID":"0459ad6c6db5719be39646515856f6a3","Type":"ContainerDied","Data":"86c55d4aea6a923581be42279421c6a7d7627e21c4599cc522d38f8b2bee8cda"} Oct 30 20:38:15.025122 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.025078 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" event={"ID":"0459ad6c6db5719be39646515856f6a3","Type":"ContainerStarted","Data":"6699dc1c4ebf7b70abd16bb5b79b34127d1801b261cf3b6e50ad3abbaee3b1db"} Oct 30 20:38:15.025122 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.025107 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal" event={"ID":"879a456e73ba77e5a50d63568935393e","Type":"ContainerStarted","Data":"f00287dbbc4430696cba50bbf839e2b51a1add1a7e83fa8ba033f5dbc6865aa3"} Oct 30 20:38:15.025184 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.025141 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal" event={"ID":"879a456e73ba77e5a50d63568935393e","Type":"ContainerStarted","Data":"b4fc79cde41391c8690bb276a888b958b0fb62293a146794e3ec73f90451e3f4"} Oct 30 20:38:15.025291 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.025277 4848 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:15.040133 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.040105 4848 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:38:15.040248 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:38:15.040157 4848 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:15.041071 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.041058 4848 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:38:15.041117 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:38:15.041081 4848 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:15.105384 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.105355 4848 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:38:15.106679 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.106663 4848 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:38:15.106758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.106694 4848 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:38:15.106758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.106712 4848 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-52.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:38:15.106758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.106755 4848 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:15.112006 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.111986 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0459ad6c6db5719be39646515856f6a3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal\" (UID: \"0459ad6c6db5719be39646515856f6a3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:15.112072 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.112013 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0459ad6c6db5719be39646515856f6a3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal\" (UID: \"0459ad6c6db5719be39646515856f6a3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:15.112072 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.112030 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/879a456e73ba77e5a50d63568935393e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal\" (UID: \"879a456e73ba77e5a50d63568935393e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:15.121646 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.121627 4848 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:15.121724 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.121683 4848 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:15.212216 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.212178 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0459ad6c6db5719be39646515856f6a3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal\" (UID: \"0459ad6c6db5719be39646515856f6a3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:15.212216 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.212218 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0459ad6c6db5719be39646515856f6a3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal\" (UID: \"0459ad6c6db5719be39646515856f6a3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:15.212508 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.212247 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/879a456e73ba77e5a50d63568935393e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal\" (UID: \"879a456e73ba77e5a50d63568935393e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:15.212508 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.212290 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0459ad6c6db5719be39646515856f6a3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal\" (UID: \"0459ad6c6db5719be39646515856f6a3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:15.212508 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.212298 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/879a456e73ba77e5a50d63568935393e-config\") pod \"kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal\" (UID: \"879a456e73ba77e5a50d63568935393e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:15.212508 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.212311 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0459ad6c6db5719be39646515856f6a3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal\" (UID: \"0459ad6c6db5719be39646515856f6a3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:15.882395 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.882366 4848 apiserver.go:52] "Watching apiserver" Oct 30 20:38:15.891238 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.891213 4848 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 20:38:15.891745 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.891724 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-554jb","openshift-multus/multus-additional-cni-plugins-n7lg7","openshift-cluster-node-tuning-operator/tuned-wr47v","openshift-dns/dns-default-tvdg8","kube-system/global-pull-secret-syncer-5wp5n","openshift-network-diagnostics/network-check-target-qs2dh","openshift-network-operator/iptables-alerter-wzxhr","openshift-dns/node-resolver-vcndb","openshift-ingress-canary/ingress-canary-9w8gw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal","openshift-multus/multus-2kvw6","kube-system/konnectivity-agent-grp6n","openshift-insights/insights-runtime-extractor-k4d59","openshift-multus/network-metrics-daemon-48zsl","kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal","openshift-monitoring/node-exporter-bqhgw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg","openshift-ovn-kubernetes/ovnkube-node-rz98l"] Oct 30 20:38:15.899096 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.899014 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:38:15.899096 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.899043 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 20:38:15.899357 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.899212 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 20:38:15.899357 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.899312 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:38:15.899471 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.899377 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 20:38:15.899471 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.899407 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 20:38:15.899794 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.899753 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-chh5h\"" Oct 30 20:38:15.900461 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.900113 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 20:38:15.900461 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.900245 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 20:38:15.900685 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.900466 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 20:38:15.900685 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.900413 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-6lk59\"" Oct 30 20:38:15.900685 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.900433 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-82bc7\"" Oct 30 20:38:15.900685 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.900357 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 20:38:15.900685 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.900593 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 20:38:15.900971 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.900696 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x97dd\"" Oct 30 20:38:15.900971 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.900821 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 20:38:15.900971 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.900835 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 20:38:15.901183 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.901008 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 20:38:15.901183 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.901060 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 20:38:15.901183 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.901080 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jrl7k\"" Oct 30 20:38:15.901183 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.901146 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 20:38:15.901482 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.901468 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4pvnz\"" Oct 30 20:38:15.903306 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.903287 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fxgqk\"" Oct 30 20:38:15.903306 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.903302 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 20:38:15.903465 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.903346 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 20:38:15.903465 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.903361 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 20:38:15.903567 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.903536 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 20:38:15.903617 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.903592 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 20:38:15.904126 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.904099 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 20:38:15.904126 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.904106 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 20:38:15.904397 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.904384 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 20:38:15.904577 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.904560 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2hx48\"" Oct 30 20:38:15.904820 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.904807 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 20:38:15.904868 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.904834 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zljsp\"" Oct 30 20:38:15.905501 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.905489 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 20:38:15.907927 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.907908 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 20:38:15.908048 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.907938 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 20:38:15.908048 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.907958 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 20:38:15.908048 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.907974 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 20:38:15.908209 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.908121 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 20:38:15.908209 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.908146 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bcgjk\"" Oct 30 20:38:15.908209 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.908151 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 20:38:15.908209 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.908177 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 20:38:15.908209 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.908185 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l4rxj\"" Oct 30 20:38:15.908209 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.908181 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 20:38:15.908556 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.908218 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5c74x\"" Oct 30 20:38:15.908556 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.908273 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7lq4m\"" Oct 30 20:38:15.908556 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.908429 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qtwx7\"" Oct 30 20:38:15.908556 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.908468 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 20:38:15.908556 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.908527 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rsx8b\"" Oct 30 20:38:15.908845 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.908628 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 20:38:15.908899 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.908893 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 20:38:15.908948 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.908908 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 20:38:15.909163 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.909145 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 20:38:15.909397 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.909383 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 20:38:15.909644 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.909627 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 20:38:15.909644 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.909639 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 20:38:15.909794 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.909662 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 20:38:15.909794 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.909682 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 20:38:15.909794 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.909733 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 20:38:15.909919 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.909907 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-69hq7\"" Oct 30 20:38:15.909994 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.909981 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 20:38:15.915270 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.915251 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-sysctl-conf\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:15.915331 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.915276 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ce76c71-8b23-49b4-a61c-9b14adea86c7-cnibin\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:38:15.915331 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.915297 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-run-netns\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:15.915331 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.915316 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-modprobe-d\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:15.915421 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.915367 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-env-overrides\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.915465 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.915415 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-ovn-node-metrics-cert\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.915465 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.915451 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-sysconfig\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:15.915589 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.915571 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-cnibin\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:15.915647 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.915599 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-run-k8s-cni-cncf-io\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:15.915647 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.915618 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-var-lib-cni-bin\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:15.915742 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.915647 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7662k\" (UniqueName: \"kubernetes.io/projected/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-kube-api-access-7662k\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.915742 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.915683 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.915742 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.915728 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-env-overrides\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.915902 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.915751 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-kubelet\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.915902 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.915795 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-slash\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.915902 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.915817 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-etc-openvswitch\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.915902 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.915833 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-ovn-node-metrics-cert\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.915902 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.915856 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-cni-bin\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.915902 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.915886 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ce76c71-8b23-49b4-a61c-9b14adea86c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:38:15.916185 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916026 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mcpr\" (UniqueName: \"kubernetes.io/projected/6ce76c71-8b23-49b4-a61c-9b14adea86c7-kube-api-access-5mcpr\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:38:15.916185 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916050 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-kubernetes\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:15.916185 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916066 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-run-openvswitch\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.916185 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916089 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ce76c71-8b23-49b4-a61c-9b14adea86c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:38:15.916185 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916115 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6ce76c71-8b23-49b4-a61c-9b14adea86c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:38:15.916354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916228 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-etc-kubernetes\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:15.916354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916248 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-run\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:15.916354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916271 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-var-lib-kubelet\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:15.916354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916287 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-tuned\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:15.916354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916314 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-log-socket\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.916354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916353 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da9ceed2-3ab8-43dc-88a3-f20d7351ac30-host\") pod \"node-ca-554jb\" (UID: \"da9ceed2-3ab8-43dc-88a3-f20d7351ac30\") " pod="openshift-image-registry/node-ca-554jb" Oct 30 20:38:15.916635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916375 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-cni-dir\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:15.916635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916389 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-tuned\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:15.916635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916396 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkqcq\" (UniqueName: \"kubernetes.io/projected/9390b293-dd28-4713-8452-b9fb5d7ccaec-kube-api-access-lkqcq\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:15.916635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916429 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-cni-netd\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.916635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916463 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-daemon-config\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:15.916635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916561 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9390b293-dd28-4713-8452-b9fb5d7ccaec-tmp\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:15.916635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916581 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-systemd-units\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.916635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916595 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-run-netns\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.916635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916609 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-var-lib-openvswitch\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.916635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916622 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9390b293-dd28-4713-8452-b9fb5d7ccaec-tmp\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:15.917225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916648 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a99706d1-d012-4dd9-b46a-5ec139287351-kubelet-config\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:38:15.917225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916680 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-run-ovn\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.917225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916710 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-ovnkube-script-lib\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.917225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916729 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-daemon-config\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:15.917225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916794 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a99706d1-d012-4dd9-b46a-5ec139287351-dbus\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:38:15.917225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916814 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-socket-dir-parent\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:15.917225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916830 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-conf-dir\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:15.917225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916844 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-systemd\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:15.917225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916869 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ce76c71-8b23-49b4-a61c-9b14adea86c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:38:15.917225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916929 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-ovnkube-script-lib\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.917225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916938 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-var-lib-cni-multus\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:15.917225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916972 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7456\" (UniqueName: \"kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456\") pod \"network-check-target-qs2dh\" (UID: \"0832230b-2d6a-40f2-8d5f-dc1da218c7b8\") " pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:38:15.917225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.916997 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j47v7\" (UniqueName: \"kubernetes.io/projected/da9ceed2-3ab8-43dc-88a3-f20d7351ac30-kube-api-access-j47v7\") pod \"node-ca-554jb\" (UID: \"da9ceed2-3ab8-43dc-88a3-f20d7351ac30\") " pod="openshift-image-registry/node-ca-554jb" Oct 30 20:38:15.917225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917010 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6ce76c71-8b23-49b4-a61c-9b14adea86c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:38:15.917225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917020 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swthh\" (UniqueName: \"kubernetes.io/projected/5b91b1fa-0886-4a2c-8005-bd6f77666874-kube-api-access-swthh\") pod \"iptables-alerter-wzxhr\" (UID: \"5b91b1fa-0886-4a2c-8005-bd6f77666874\") " pod="openshift-network-operator/iptables-alerter-wzxhr" Oct 30 20:38:15.917225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917044 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-system-cni-dir\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:15.917225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917163 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0137d1fc-8fa3-4b0e-8821-4776913088f7-cni-binary-copy\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:15.917225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917199 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-var-lib-kubelet\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:15.917815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917260 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-hostroot\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:15.917815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917291 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-run-ovn-kubernetes\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.917815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917321 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-ovnkube-config\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.917815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917387 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0137d1fc-8fa3-4b0e-8821-4776913088f7-cni-binary-copy\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:15.917815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917431 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b91b1fa-0886-4a2c-8005-bd6f77666874-host-slash\") pod \"iptables-alerter-wzxhr\" (UID: \"5b91b1fa-0886-4a2c-8005-bd6f77666874\") " pod="openshift-network-operator/iptables-alerter-wzxhr" Oct 30 20:38:15.917815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917450 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-sysctl-d\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:15.917815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917470 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:38:15.917815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917498 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da9ceed2-3ab8-43dc-88a3-f20d7351ac30-serviceca\") pod \"node-ca-554jb\" (UID: \"da9ceed2-3ab8-43dc-88a3-f20d7351ac30\") " pod="openshift-image-registry/node-ca-554jb" Oct 30 20:38:15.917815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917501 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-ovnkube-config\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.917815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917532 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbdm6\" (UniqueName: \"kubernetes.io/projected/0137d1fc-8fa3-4b0e-8821-4776913088f7-kube-api-access-rbdm6\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:15.917815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917653 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-host\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:15.917815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917717 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a99706d1-d012-4dd9-b46a-5ec139287351-original-pull-secret\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:38:15.917815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917753 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-run-systemd\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.917815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917786 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ce76c71-8b23-49b4-a61c-9b14adea86c7-system-cni-dir\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:38:15.917815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917807 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ce76c71-8b23-49b4-a61c-9b14adea86c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:38:15.918326 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917801 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da9ceed2-3ab8-43dc-88a3-f20d7351ac30-serviceca\") pod \"node-ca-554jb\" (UID: \"da9ceed2-3ab8-43dc-88a3-f20d7351ac30\") " pod="openshift-image-registry/node-ca-554jb" Oct 30 20:38:15.918326 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917839 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-run-multus-certs\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:15.918326 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917950 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-lib-modules\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:15.918326 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917980 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6ce76c71-8b23-49b4-a61c-9b14adea86c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:38:15.918326 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.917987 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b91b1fa-0886-4a2c-8005-bd6f77666874-iptables-alerter-script\") pod \"iptables-alerter-wzxhr\" (UID: \"5b91b1fa-0886-4a2c-8005-bd6f77666874\") " pod="openshift-network-operator/iptables-alerter-wzxhr" Oct 30 20:38:15.918326 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.918031 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-sys\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:15.918326 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.918173 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-node-log\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.918326 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.918213 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b91b1fa-0886-4a2c-8005-bd6f77666874-iptables-alerter-script\") pod \"iptables-alerter-wzxhr\" (UID: \"5b91b1fa-0886-4a2c-8005-bd6f77666874\") " pod="openshift-network-operator/iptables-alerter-wzxhr" Oct 30 20:38:15.924015 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.923992 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7456\" (UniqueName: \"kubernetes.io/projected/0832230b-2d6a-40f2-8d5f-dc1da218c7b8-kube-api-access-v7456\") pod \"network-check-target-qs2dh\" (UID: \"0832230b-2d6a-40f2-8d5f-dc1da218c7b8\") " pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:38:15.925115 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.925095 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mcpr\" (UniqueName: \"kubernetes.io/projected/6ce76c71-8b23-49b4-a61c-9b14adea86c7-kube-api-access-5mcpr\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:38:15.925340 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.925322 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j47v7\" (UniqueName: \"kubernetes.io/projected/da9ceed2-3ab8-43dc-88a3-f20d7351ac30-kube-api-access-j47v7\") pod \"node-ca-554jb\" (UID: \"da9ceed2-3ab8-43dc-88a3-f20d7351ac30\") " pod="openshift-image-registry/node-ca-554jb" Oct 30 20:38:15.925456 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.925435 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkqcq\" (UniqueName: \"kubernetes.io/projected/9390b293-dd28-4713-8452-b9fb5d7ccaec-kube-api-access-lkqcq\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:15.927163 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.927144 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7662k\" (UniqueName: \"kubernetes.io/projected/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-kube-api-access-7662k\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:15.927291 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.927275 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbdm6\" (UniqueName: \"kubernetes.io/projected/0137d1fc-8fa3-4b0e-8821-4776913088f7-kube-api-access-rbdm6\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:15.928025 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.928007 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swthh\" (UniqueName: \"kubernetes.io/projected/5b91b1fa-0886-4a2c-8005-bd6f77666874-kube-api-access-swthh\") pod \"iptables-alerter-wzxhr\" (UID: \"5b91b1fa-0886-4a2c-8005-bd6f77666874\") " pod="openshift-network-operator/iptables-alerter-wzxhr" Oct 30 20:38:15.972414 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.972387 4848 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:15.972554 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.972531 4848 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:15.985808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.985785 4848 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:38:15.985885 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:38:15.985831 4848 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:15.986507 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.986495 4848 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:38:15.986544 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:38:15.986524 4848 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-52.us-east-2.compute.internal" Oct 30 20:38:15.995429 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:15.995406 4848 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 20:38:16.019463 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019433 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7ns5\" (UniqueName: \"kubernetes.io/projected/a8e2605f-358c-4fd1-ae8a-524282c76f72-kube-api-access-s7ns5\") pod \"ingress-canary-9w8gw\" (UID: \"a8e2605f-358c-4fd1-ae8a-524282c76f72\") " pod="openshift-ingress-canary/ingress-canary-9w8gw" Oct 30 20:38:16.019596 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019470 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-registration-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:38:16.019596 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019495 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da9ceed2-3ab8-43dc-88a3-f20d7351ac30-host\") pod \"node-ca-554jb\" (UID: \"da9ceed2-3ab8-43dc-88a3-f20d7351ac30\") " pod="openshift-image-registry/node-ca-554jb" Oct 30 20:38:16.019596 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019515 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-cni-dir\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.019596 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019531 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-cni-netd\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.019596 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019581 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c57ecae-89cb-4901-8f9e-e4586771f5d8-agent-certs\") pod \"konnectivity-agent-grp6n\" (UID: \"4c57ecae-89cb-4901-8f9e-e4586771f5d8\") " pod="kube-system/konnectivity-agent-grp6n" Oct 30 20:38:16.019874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019596 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da9ceed2-3ab8-43dc-88a3-f20d7351ac30-host\") pod \"node-ca-554jb\" (UID: \"da9ceed2-3ab8-43dc-88a3-f20d7351ac30\") " pod="openshift-image-registry/node-ca-554jb" Oct 30 20:38:16.019874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019613 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-sys-fs\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:38:16.019874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019641 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-systemd-units\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.019874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019658 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-run-netns\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.019874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019676 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-var-lib-openvswitch\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.019874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019685 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-cni-dir\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.019874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019701 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a99706d1-d012-4dd9-b46a-5ec139287351-kubelet-config\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:38:16.019874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019720 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-cni-netd\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.019874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019727 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm59l\" (UniqueName: \"kubernetes.io/projected/a97f29d4-5d15-432a-a493-6011cf8e059e-kube-api-access-dm59l\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.019874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019803 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-socket-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:38:16.019874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019836 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-run-netns\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.019874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019848 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-run-ovn\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.019874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019866 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a99706d1-d012-4dd9-b46a-5ec139287351-kubelet-config\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:38:16.019874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019865 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-systemd-units\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.020614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019902 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-run-ovn\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.020614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019920 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a99706d1-d012-4dd9-b46a-5ec139287351-dbus\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:38:16.020614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.019992 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a99706d1-d012-4dd9-b46a-5ec139287351-dbus\") pod \"global-pull-secret-syncer-5wp5n\" (UID: \"a99706d1-d012-4dd9-b46a-5ec139287351\") " pod="kube-system/global-pull-secret-syncer-5wp5n" Oct 30 20:38:16.020614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.020036 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b66dabb7-8606-4898-99ca-954ffd602da2-metrics-tls\") pod \"dns-default-tvdg8\" (UID: \"b66dabb7-8606-4898-99ca-954ffd602da2\") " pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:16.020614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.020046 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-var-lib-openvswitch\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.020614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.020090 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4c57ecae-89cb-4901-8f9e-e4586771f5d8-agent-certs\") pod \"konnectivity-agent-grp6n\" (UID: \"4c57ecae-89cb-4901-8f9e-e4586771f5d8\") " pod="kube-system/konnectivity-agent-grp6n" Oct 30 20:38:16.020614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.020095 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-socket-dir-parent\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.020614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.020220 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-conf-dir\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.020614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.020300 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-systemd\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.020614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.020347 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b66dabb7-8606-4898-99ca-954ffd602da2-metrics-tls\") pod \"dns-default-tvdg8\" (UID: \"b66dabb7-8606-4898-99ca-954ffd602da2\") " pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:16.020614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.020366 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-conf-dir\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.020614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.020414 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-multus-socket-dir-parent\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.020614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.020471 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-systemd\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.020614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.020503 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-accelerators-collector-config\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.020614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.020537 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b66dabb7-8606-4898-99ca-954ffd602da2-tmp-dir\") pod \"dns-default-tvdg8\" (UID: \"b66dabb7-8606-4898-99ca-954ffd602da2\") " pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:16.020614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.020560 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-var-lib-cni-multus\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.021373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.020861 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-accelerators-collector-config\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.021373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.020902 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-tls\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.021373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.020929 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-wtmp\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.021373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.020974 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a97f29d4-5d15-432a-a493-6011cf8e059e-metrics-client-ca\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.021373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.020996 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slwwv\" (UniqueName: \"kubernetes.io/projected/dbe199fd-1f79-4fa3-9136-21a58fa1202d-kube-api-access-slwwv\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:16.021373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021025 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-system-cni-dir\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.021373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021070 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-var-lib-kubelet\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.021373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021088 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-hostroot\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.021373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021106 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-run-ovn-kubernetes\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.021373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021122 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a97f29d4-5d15-432a-a493-6011cf8e059e-sys\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.021373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021138 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b91b1fa-0886-4a2c-8005-bd6f77666874-host-slash\") pod \"iptables-alerter-wzxhr\" (UID: \"5b91b1fa-0886-4a2c-8005-bd6f77666874\") " pod="openshift-network-operator/iptables-alerter-wzxhr" Oct 30 20:38:16.021373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021140 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b66dabb7-8606-4898-99ca-954ffd602da2-tmp-dir\") pod \"dns-default-tvdg8\" (UID: \"b66dabb7-8606-4898-99ca-954ffd602da2\") " pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:16.021373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021196 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-hostroot\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.021373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021220 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-system-cni-dir\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.021373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021249 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-sysctl-d\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.021373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021277 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.021373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021306 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dbe199fd-1f79-4fa3-9136-21a58fa1202d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:16.021373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021355 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs\") pod \"network-metrics-daemon-48zsl\" (UID: \"51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249\") " pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:38:16.021373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021386 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:38:16.022367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021371 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-run-ovn-kubernetes\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.022367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021427 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-host\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.022367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021386 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b91b1fa-0886-4a2c-8005-bd6f77666874-host-slash\") pod \"iptables-alerter-wzxhr\" (UID: \"5b91b1fa-0886-4a2c-8005-bd6f77666874\") " pod="openshift-network-operator/iptables-alerter-wzxhr" Oct 30 20:38:16.022367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021463 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-tls\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.022367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021495 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-host\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.022367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021558 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-var-lib-cni-multus\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.022367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021665 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-run-systemd\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.022367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021696 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-var-lib-kubelet\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.022367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021711 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dbe199fd-1f79-4fa3-9136-21a58fa1202d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:16.022367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021759 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ce76c71-8b23-49b4-a61c-9b14adea86c7-system-cni-dir\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:38:16.022367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021874 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.022367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021889 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a97f29d4-5d15-432a-a493-6011cf8e059e-metrics-client-ca\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.022367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021951 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-metrics-certs\") pod \"network-metrics-daemon-48zsl\" (UID: \"51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249\") " pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:38:16.022367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021949 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-run-systemd\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.022367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.021967 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bsb9\" (UniqueName: \"kubernetes.io/projected/313047e7-bc02-42eb-933b-ab52575af3f7-kube-api-access-9bsb9\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:38:16.022367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.022050 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-run-multus-certs\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.023231 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.022074 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-sysctl-d\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.023231 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.022090 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6ce76c71-8b23-49b4-a61c-9b14adea86c7-system-cni-dir\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:38:16.023231 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.022197 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-run-multus-certs\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.023231 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.023157 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-lib-modules\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.023452 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.023228 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab625ab1-de4f-4418-bd65-edb628d963fe-tmp-dir\") pod \"node-resolver-vcndb\" (UID: \"ab625ab1-de4f-4418-bd65-edb628d963fe\") " pod="openshift-dns/node-resolver-vcndb" Oct 30 20:38:16.023452 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.023370 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-sys\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.023551 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.023452 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-node-log\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.023551 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.023514 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a97f29d4-5d15-432a-a493-6011cf8e059e-root\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.023651 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.023585 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dbe199fd-1f79-4fa3-9136-21a58fa1202d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:16.023725 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.023686 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v69k5\" (UniqueName: \"kubernetes.io/projected/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-kube-api-access-v69k5\") pod \"network-metrics-daemon-48zsl\" (UID: \"51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249\") " pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:38:16.023911 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.023791 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab625ab1-de4f-4418-bd65-edb628d963fe-hosts-file\") pod \"node-resolver-vcndb\" (UID: \"ab625ab1-de4f-4418-bd65-edb628d963fe\") " pod="openshift-dns/node-resolver-vcndb" Oct 30 20:38:16.023911 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.023812 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-sys\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.023911 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.023826 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqs5f\" (UniqueName: \"kubernetes.io/projected/b66dabb7-8606-4898-99ca-954ffd602da2-kube-api-access-mqs5f\") pod \"dns-default-tvdg8\" (UID: \"b66dabb7-8606-4898-99ca-954ffd602da2\") " pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:16.024060 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.023932 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab625ab1-de4f-4418-bd65-edb628d963fe-tmp-dir\") pod \"node-resolver-vcndb\" (UID: \"ab625ab1-de4f-4418-bd65-edb628d963fe\") " pod="openshift-dns/node-resolver-vcndb" Oct 30 20:38:16.024060 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.023986 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-lib-modules\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.024178 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024160 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-sysctl-conf\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.024260 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024192 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ce76c71-8b23-49b4-a61c-9b14adea86c7-cnibin\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:38:16.024260 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024216 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbe199fd-1f79-4fa3-9136-21a58fa1202d-data-volume\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:16.024260 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024241 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b66dabb7-8606-4898-99ca-954ffd602da2-config-volume\") pod \"dns-default-tvdg8\" (UID: \"b66dabb7-8606-4898-99ca-954ffd602da2\") " pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:16.024408 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024275 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-run-netns\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.024408 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024304 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-modprobe-d\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.024408 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024347 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-textfile\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.024408 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024359 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-node-log\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.024408 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024373 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:38:16.024408 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024402 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-sysconfig\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.024649 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024416 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-run-netns\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.024649 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024425 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-cnibin\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.024649 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024462 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-run-k8s-cni-cncf-io\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.024649 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024488 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-sysctl-conf\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.024649 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024530 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6ce76c71-8b23-49b4-a61c-9b14adea86c7-cnibin\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:38:16.024649 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024540 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-modprobe-d\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.024649 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024542 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-var-lib-cni-bin\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.024649 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024492 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-var-lib-cni-bin\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.024649 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024604 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbe199fd-1f79-4fa3-9136-21a58fa1202d-data-volume\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:16.024649 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024634 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-cnibin\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.025010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024668 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-textfile\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.025010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024695 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7krsp\" (UniqueName: \"kubernetes.io/projected/ab625ab1-de4f-4418-bd65-edb628d963fe-kube-api-access-7krsp\") pod \"node-resolver-vcndb\" (UID: \"ab625ab1-de4f-4418-bd65-edb628d963fe\") " pod="openshift-dns/node-resolver-vcndb" Oct 30 20:38:16.025010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024702 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-sysconfig\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.025010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024797 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.025010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024826 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-kubelet\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.025010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024851 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-slash\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.025010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024891 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-etc-openvswitch\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.025010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024909 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-cni-bin\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.025010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024929 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c57ecae-89cb-4901-8f9e-e4586771f5d8-konnectivity-ca\") pod \"konnectivity-agent-grp6n\" (UID: \"4c57ecae-89cb-4901-8f9e-e4586771f5d8\") " pod="kube-system/konnectivity-agent-grp6n" Oct 30 20:38:16.025010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024963 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-slash\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.025010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.024997 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-host-run-k8s-cni-cncf-io\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.025370 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025031 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.025370 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025051 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-kubelet\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.025370 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025061 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b66dabb7-8606-4898-99ca-954ffd602da2-config-volume\") pod \"dns-default-tvdg8\" (UID: \"b66dabb7-8606-4898-99ca-954ffd602da2\") " pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:16.025370 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025073 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-etc-openvswitch\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.025370 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025094 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-host-cni-bin\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.025370 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025114 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-kubernetes\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.025370 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025200 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-run-openvswitch\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.025370 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025215 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ce76c71-8b23-49b4-a61c-9b14adea86c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:38:16.025370 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025244 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8e2605f-358c-4fd1-ae8a-524282c76f72-cert\") pod \"ingress-canary-9w8gw\" (UID: \"a8e2605f-358c-4fd1-ae8a-524282c76f72\") " pod="openshift-ingress-canary/ingress-canary-9w8gw" Oct 30 20:38:16.025370 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025283 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dbe199fd-1f79-4fa3-9136-21a58fa1202d-crio-socket\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:16.025370 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025303 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-device-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:38:16.025370 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025319 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-etc-kubernetes\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.025370 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025333 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-run\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.025370 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025349 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-var-lib-kubelet\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.026010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025514 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a8e2605f-358c-4fd1-ae8a-524282c76f72-cert\") pod \"ingress-canary-9w8gw\" (UID: \"a8e2605f-358c-4fd1-ae8a-524282c76f72\") " pod="openshift-ingress-canary/ingress-canary-9w8gw" Oct 30 20:38:16.026010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025546 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-etc-kubernetes\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.026010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025552 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4c57ecae-89cb-4901-8f9e-e4586771f5d8-konnectivity-ca\") pod \"konnectivity-agent-grp6n\" (UID: \"4c57ecae-89cb-4901-8f9e-e4586771f5d8\") " pod="kube-system/konnectivity-agent-grp6n" Oct 30 20:38:16.026010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025702 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0137d1fc-8fa3-4b0e-8821-4776913088f7-etc-kubernetes\") pod \"multus-2kvw6\" (UID: \"0137d1fc-8fa3-4b0e-8821-4776913088f7\") " pod="openshift-multus/multus-2kvw6" Oct 30 20:38:16.026010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025817 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-log-socket\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.026010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025843 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6ce76c71-8b23-49b4-a61c-9b14adea86c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n7lg7\" (UID: \"6ce76c71-8b23-49b4-a61c-9b14adea86c7\") " pod="openshift-multus/multus-additional-cni-plugins-n7lg7" Oct 30 20:38:16.026010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025854 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-run-openvswitch\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.026010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025883 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9224f6c0-bc3e-4eee-a78b-431e6c8f4705-log-socket\") pod \"ovnkube-node-rz98l\" (UID: \"9224f6c0-bc3e-4eee-a78b-431e6c8f4705\") " pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:16.026010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.025956 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-run\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.026440 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.026370 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9390b293-dd28-4713-8452-b9fb5d7ccaec-var-lib-kubelet\") pod \"tuned-wr47v\" (UID: \"9390b293-dd28-4713-8452-b9fb5d7ccaec\") " pod="openshift-cluster-node-tuning-operator/tuned-wr47v" Oct 30 20:38:16.026600 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.026581 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dbe199fd-1f79-4fa3-9136-21a58fa1202d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:16.026790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.026755 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7ns5\" (UniqueName: \"kubernetes.io/projected/a8e2605f-358c-4fd1-ae8a-524282c76f72-kube-api-access-s7ns5\") pod \"ingress-canary-9w8gw\" (UID: \"a8e2605f-358c-4fd1-ae8a-524282c76f72\") " pod="openshift-ingress-canary/ingress-canary-9w8gw" Oct 30 20:38:16.026857 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.026820 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm59l\" (UniqueName: \"kubernetes.io/projected/a97f29d4-5d15-432a-a493-6011cf8e059e-kube-api-access-dm59l\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.028742 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.028727 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slwwv\" (UniqueName: \"kubernetes.io/projected/dbe199fd-1f79-4fa3-9136-21a58fa1202d-kube-api-access-slwwv\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:16.029109 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.029089 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bsb9\" (UniqueName: \"kubernetes.io/projected/313047e7-bc02-42eb-933b-ab52575af3f7-kube-api-access-9bsb9\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:38:16.030089 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.030071 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v69k5\" (UniqueName: \"kubernetes.io/projected/51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249-kube-api-access-v69k5\") pod \"network-metrics-daemon-48zsl\" (UID: \"51eb0ba7-2dfd-471f-98e1-5dcdb1ab2249\") " pod="openshift-multus/network-metrics-daemon-48zsl" Oct 30 20:38:16.030495 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.030479 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqs5f\" (UniqueName: \"kubernetes.io/projected/b66dabb7-8606-4898-99ca-954ffd602da2-kube-api-access-mqs5f\") pod \"dns-default-tvdg8\" (UID: \"b66dabb7-8606-4898-99ca-954ffd602da2\") " pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:16.031852 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.031838 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7krsp\" (UniqueName: \"kubernetes.io/projected/ab625ab1-de4f-4418-bd65-edb628d963fe-kube-api-access-7krsp\") pod \"node-resolver-vcndb\" (UID: \"ab625ab1-de4f-4418-bd65-edb628d963fe\") " pod="openshift-dns/node-resolver-vcndb" Oct 30 20:38:16.126556 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126527 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-registration-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:38:16.126556 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126561 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-sys-fs\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:38:16.126738 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126579 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-socket-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:38:16.126738 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126653 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-socket-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:38:16.126738 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126661 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-wtmp\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.126738 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126665 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-registration-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:38:16.126738 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126670 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-sys-fs\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:38:16.126738 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126705 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a97f29d4-5d15-432a-a493-6011cf8e059e-sys\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.126738 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126732 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a97f29d4-5d15-432a-a493-6011cf8e059e-sys\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.127015 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126759 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:38:16.127015 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126791 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a97f29d4-5d15-432a-a493-6011cf8e059e-node-exporter-wtmp\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.127015 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126809 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a97f29d4-5d15-432a-a493-6011cf8e059e-root\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.127015 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126820 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:38:16.127015 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126836 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab625ab1-de4f-4418-bd65-edb628d963fe-hosts-file\") pod \"node-resolver-vcndb\" (UID: \"ab625ab1-de4f-4418-bd65-edb628d963fe\") " pod="openshift-dns/node-resolver-vcndb" Oct 30 20:38:16.127015 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126866 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:38:16.127015 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126898 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dbe199fd-1f79-4fa3-9136-21a58fa1202d-crio-socket\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:16.127015 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126907 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a97f29d4-5d15-432a-a493-6011cf8e059e-root\") pod \"node-exporter-bqhgw\" (UID: \"a97f29d4-5d15-432a-a493-6011cf8e059e\") " pod="openshift-monitoring/node-exporter-bqhgw" Oct 30 20:38:16.127015 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126919 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-device-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:38:16.127015 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126929 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab625ab1-de4f-4418-bd65-edb628d963fe-hosts-file\") pod \"node-resolver-vcndb\" (UID: \"ab625ab1-de4f-4418-bd65-edb628d963fe\") " pod="openshift-dns/node-resolver-vcndb" Oct 30 20:38:16.127015 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126955 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dbe199fd-1f79-4fa3-9136-21a58fa1202d-crio-socket\") pod \"insights-runtime-extractor-k4d59\" (UID: \"dbe199fd-1f79-4fa3-9136-21a58fa1202d\") " pod="openshift-insights/insights-runtime-extractor-k4d59" Oct 30 20:38:16.127015 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126946 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-etc-selinux\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:38:16.127015 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:16.126987 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/313047e7-bc02-42eb-933b-ab52575af3f7-device-dir\") pod \"aws-ebs-csi-driver-node-kh5tg\" (UID: \"313047e7-bc02-42eb-933b-ab52575af3f7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kh5tg" Oct 30 20:38:17.221637 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:17.221598 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:17.223296 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:17.223273 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tvdg8" Oct 30 20:38:17.497167 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:17.497082 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:38:20.674274 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:20.674247 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:22.508413 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:38:22.508371 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:38:25.248995 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:25.248966 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:25.249443 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:25.249220 4848 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:38:35.825651 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:35.825620 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-grp6n" Oct 30 20:38:45.561480 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.561326 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-59d56c645b-72gc5"] Oct 30 20:38:45.591395 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.591370 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59d56c645b-72gc5"] Oct 30 20:38:45.591549 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.591461 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.595976 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.595948 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 20:38:45.597218 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.597069 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 20:38:45.597218 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.597069 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ppxb4\"" Oct 30 20:38:45.597218 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.597096 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 20:38:45.597218 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.597072 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 20:38:45.597218 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.597130 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 20:38:45.597218 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.597185 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 20:38:45.597218 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.597202 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 20:38:45.600370 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.600350 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 20:38:45.706000 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.705967 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f311652d-e8bf-4256-87cb-ec157cdf73c3-console-oauth-config\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.706000 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.705999 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f311652d-e8bf-4256-87cb-ec157cdf73c3-service-ca\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.706190 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.706016 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f311652d-e8bf-4256-87cb-ec157cdf73c3-oauth-serving-cert\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.706190 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.706041 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f311652d-e8bf-4256-87cb-ec157cdf73c3-console-config\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.706190 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.706150 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f311652d-e8bf-4256-87cb-ec157cdf73c3-tmp\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.706190 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.706179 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlg9m\" (UniqueName: \"kubernetes.io/projected/f311652d-e8bf-4256-87cb-ec157cdf73c3-kube-api-access-zlg9m\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.706324 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.706203 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f311652d-e8bf-4256-87cb-ec157cdf73c3-console-serving-cert\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.706324 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.706246 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f311652d-e8bf-4256-87cb-ec157cdf73c3-trusted-ca-bundle\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.807116 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.807088 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f311652d-e8bf-4256-87cb-ec157cdf73c3-console-oauth-config\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.807234 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.807120 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f311652d-e8bf-4256-87cb-ec157cdf73c3-service-ca\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.807234 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.807135 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f311652d-e8bf-4256-87cb-ec157cdf73c3-oauth-serving-cert\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.807234 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.807151 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f311652d-e8bf-4256-87cb-ec157cdf73c3-console-config\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.807353 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.807290 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f311652d-e8bf-4256-87cb-ec157cdf73c3-tmp\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.807353 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.807338 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlg9m\" (UniqueName: \"kubernetes.io/projected/f311652d-e8bf-4256-87cb-ec157cdf73c3-kube-api-access-zlg9m\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.807461 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.807385 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f311652d-e8bf-4256-87cb-ec157cdf73c3-console-serving-cert\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.807461 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.807423 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f311652d-e8bf-4256-87cb-ec157cdf73c3-trusted-ca-bundle\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.807569 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.807531 4848 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 20:38:45.807702 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.807683 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f311652d-e8bf-4256-87cb-ec157cdf73c3-tmp\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.808006 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.807978 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f311652d-e8bf-4256-87cb-ec157cdf73c3-service-ca\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.808006 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.807994 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f311652d-e8bf-4256-87cb-ec157cdf73c3-oauth-serving-cert\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.808166 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.807984 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f311652d-e8bf-4256-87cb-ec157cdf73c3-console-config\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.808232 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.808217 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f311652d-e8bf-4256-87cb-ec157cdf73c3-trusted-ca-bundle\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.809855 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.809836 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f311652d-e8bf-4256-87cb-ec157cdf73c3-console-oauth-config\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.809855 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.809842 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f311652d-e8bf-4256-87cb-ec157cdf73c3-console-serving-cert\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.816665 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.816618 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlg9m\" (UniqueName: \"kubernetes.io/projected/f311652d-e8bf-4256-87cb-ec157cdf73c3-kube-api-access-zlg9m\") pod \"console-59d56c645b-72gc5\" (UID: \"f311652d-e8bf-4256-87cb-ec157cdf73c3\") " pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:45.915703 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:45.915674 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:46.045017 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:46.044991 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rz98l" Oct 30 20:38:46.214506 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:46.214470 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59d56c645b-72gc5"] Oct 30 20:38:46.219942 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:46.217870 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf311652d_e8bf_4256_87cb_ec157cdf73c3.slice/crio-fae16e1a1e7b8c73e71781298bb2a936ff52859b0a65907f32c1fc79df27aba3 WatchSource:0}: Error finding container fae16e1a1e7b8c73e71781298bb2a936ff52859b0a65907f32c1fc79df27aba3: Status 404 returned error can't find the container with id fae16e1a1e7b8c73e71781298bb2a936ff52859b0a65907f32c1fc79df27aba3 Oct 30 20:38:46.221013 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:46.220993 4848 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:38:47.063231 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:47.063188 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d56c645b-72gc5" event={"ID":"f311652d-e8bf-4256-87cb-ec157cdf73c3","Type":"ContainerStarted","Data":"fae16e1a1e7b8c73e71781298bb2a936ff52859b0a65907f32c1fc79df27aba3"} Oct 30 20:38:47.500627 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:47.500596 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qs2dh" Oct 30 20:38:48.879515 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:48.879455 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 20:38:48.884999 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:48.884974 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:48.889558 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:48.889531 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 20:38:48.889558 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:48.889549 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 20:38:48.889558 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:48.889549 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 20:38:48.889813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:48.889531 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 20:38:48.890334 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:48.890312 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 20:38:48.890334 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:48.890314 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 20:38:48.890688 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:48.890672 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-92xsf\"" Oct 30 20:38:48.890753 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:48.890691 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 20:38:48.890821 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:48.890780 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 20:38:48.892759 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:48.892742 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 20:38:48.904041 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:48.904019 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 20:38:49.031916 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.031883 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.031916 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.031918 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.032159 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.031943 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b8fa08e-4def-48af-9546-903b17761894-config-out\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.032159 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.031965 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.032159 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.032137 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b8fa08e-4def-48af-9546-903b17761894-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.032291 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.032189 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b8fa08e-4def-48af-9546-903b17761894-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.032291 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.032246 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znstv\" (UniqueName: \"kubernetes.io/projected/6b8fa08e-4def-48af-9546-903b17761894-kube-api-access-znstv\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.032291 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.032279 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.032380 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.032329 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b8fa08e-4def-48af-9546-903b17761894-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.032380 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.032354 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-web-config\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.032380 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.032371 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.032464 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.032440 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-config-volume\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.032464 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.032459 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b8fa08e-4def-48af-9546-903b17761894-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.072418 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.072382 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d56c645b-72gc5" event={"ID":"f311652d-e8bf-4256-87cb-ec157cdf73c3","Type":"ContainerStarted","Data":"5b1acc7f633ad17c397a4d47308dcffefd9d338fa95fbb07e6673dc78565f682"} Oct 30 20:38:49.096779 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.096711 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-59d56c645b-72gc5" podStartSLOduration=1.397058237 podStartE2EDuration="4.096693616s" totalImagesPullingTime="2.699635379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:49.095449652 +0000 UTC m=+34.275346159" watchObservedRunningTime="2026-10-30 20:38:49.096693616 +0000 UTC m=+34.276590124" Oct 30 20:38:49.133291 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.133203 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b8fa08e-4def-48af-9546-903b17761894-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.133291 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.133247 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b8fa08e-4def-48af-9546-903b17761894-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.133291 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.133267 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znstv\" (UniqueName: \"kubernetes.io/projected/6b8fa08e-4def-48af-9546-903b17761894-kube-api-access-znstv\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.133291 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.133288 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.133631 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.133311 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b8fa08e-4def-48af-9546-903b17761894-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.133631 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.133341 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-web-config\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.133631 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.133364 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.133631 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.133394 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-config-volume\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.133631 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.133417 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b8fa08e-4def-48af-9546-903b17761894-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.133631 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.133474 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.133631 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.133497 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.133631 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.133527 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b8fa08e-4def-48af-9546-903b17761894-config-out\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.133631 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.133552 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.134112 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.134005 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6b8fa08e-4def-48af-9546-903b17761894-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.134175 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.134133 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b8fa08e-4def-48af-9546-903b17761894-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.134288 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.134264 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6b8fa08e-4def-48af-9546-903b17761894-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.137053 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.136820 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6b8fa08e-4def-48af-9546-903b17761894-config-out\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.137053 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.136870 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.137053 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.136939 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.137053 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.136998 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.137276 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.137233 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-config-volume\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.137312 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.137281 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.137532 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.137514 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6b8fa08e-4def-48af-9546-903b17761894-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.137566 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.137549 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-web-config\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.138043 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.138027 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6b8fa08e-4def-48af-9546-903b17761894-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.142880 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.142859 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znstv\" (UniqueName: \"kubernetes.io/projected/6b8fa08e-4def-48af-9546-903b17761894-kube-api-access-znstv\") pod \"alertmanager-main-0\" (UID: \"6b8fa08e-4def-48af-9546-903b17761894\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.202334 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.202292 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 20:38:49.482289 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:49.482189 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 20:38:49.486648 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:38:49.486625 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8fa08e_4def_48af_9546_903b17761894.slice/crio-e03fb545dd6c3f947f51f23d484132fd3a73c3aa6ce77452af34a1fb57a61944 WatchSource:0}: Error finding container e03fb545dd6c3f947f51f23d484132fd3a73c3aa6ce77452af34a1fb57a61944: Status 404 returned error can't find the container with id e03fb545dd6c3f947f51f23d484132fd3a73c3aa6ce77452af34a1fb57a61944 Oct 30 20:38:50.077429 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:50.077391 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b8fa08e-4def-48af-9546-903b17761894","Type":"ContainerStarted","Data":"e03fb545dd6c3f947f51f23d484132fd3a73c3aa6ce77452af34a1fb57a61944"} Oct 30 20:38:51.081871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:51.081830 4848 generic.go:356] "Generic (PLEG): container finished" podID="6b8fa08e-4def-48af-9546-903b17761894" containerID="41dacca7f501fdcb143a2b402e5fadf8f73a47197bca164fcdffbe3fd7a8bb71" exitCode=0 Oct 30 20:38:51.082335 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:51.081891 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b8fa08e-4def-48af-9546-903b17761894","Type":"ContainerDied","Data":"41dacca7f501fdcb143a2b402e5fadf8f73a47197bca164fcdffbe3fd7a8bb71"} Oct 30 20:38:52.509778 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:38:52.509741 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:38:53.090383 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:53.090348 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b8fa08e-4def-48af-9546-903b17761894","Type":"ContainerStarted","Data":"0ba20ffa42b6e5275dd7a13d62886d236b132e0595349da363276a44a1450991"} Oct 30 20:38:53.090552 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:53.090411 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b8fa08e-4def-48af-9546-903b17761894","Type":"ContainerStarted","Data":"9fca4385edc7b37c82302d7de953cad09302e6ca7369a1acc8d4f5edf22ae7fd"} Oct 30 20:38:53.090552 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:53.090437 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b8fa08e-4def-48af-9546-903b17761894","Type":"ContainerStarted","Data":"a15b15d21002b8579bcf8b846ddbe12dcaf0738b9753a1ecdcf92d520796e058"} Oct 30 20:38:53.090552 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:53.090459 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b8fa08e-4def-48af-9546-903b17761894","Type":"ContainerStarted","Data":"c721a1f9852365fb0fb4ec64810b966fc91d46a3f4e2874953a5b1e0add9a04a"} Oct 30 20:38:53.090552 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:53.090482 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b8fa08e-4def-48af-9546-903b17761894","Type":"ContainerStarted","Data":"530123d245cdbbf1192265cc193fb6f4e6ee6b6a77110856b0ce7d5d6b166874"} Oct 30 20:38:54.096634 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:54.096597 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6b8fa08e-4def-48af-9546-903b17761894","Type":"ContainerStarted","Data":"0773233919d510d5fa340af1218990b4541a4a64843b278379585ffe74b6e040"} Oct 30 20:38:54.123749 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:54.123706 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.151987041 podStartE2EDuration="6.123694302s" totalImagesPullingTime="2.958435396s" totalInitContainerRuntime="13.271865ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:38:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:38:54.122503923 +0000 UTC m=+39.302400433" watchObservedRunningTime="2026-10-30 20:38:54.123694302 +0000 UTC m=+39.303590810" Oct 30 20:38:55.916023 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:55.915974 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:55.916416 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:55.916056 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:55.920827 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:55.920805 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:38:56.110549 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:38:56.110524 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59d56c645b-72gc5" Oct 30 20:39:22.508223 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:39:22.508186 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:39:52.508693 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:39:52.508651 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:40:22.508488 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:40:22.508404 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:40:52.507871 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:40:52.507835 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:41:00.850894 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:00.850860 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5"] Oct 30 20:41:00.854133 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:00.854116 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5" Oct 30 20:41:00.863050 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:00.863031 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 20:41:00.863264 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:00.863248 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 20:41:00.864181 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:00.864167 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-6gthp\"" Oct 30 20:41:00.866685 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:00.866666 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5"] Oct 30 20:41:01.022690 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.022650 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5\" (UID: \"f8cb80d6-3ab3-4cee-90df-745aa3f33e1b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5" Oct 30 20:41:01.022884 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.022705 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5\" (UID: \"f8cb80d6-3ab3-4cee-90df-745aa3f33e1b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5" Oct 30 20:41:01.022884 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.022840 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g24f\" (UniqueName: \"kubernetes.io/projected/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-kube-api-access-8g24f\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5\" (UID: \"f8cb80d6-3ab3-4cee-90df-745aa3f33e1b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5" Oct 30 20:41:01.123421 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.123338 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g24f\" (UniqueName: \"kubernetes.io/projected/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-kube-api-access-8g24f\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5\" (UID: \"f8cb80d6-3ab3-4cee-90df-745aa3f33e1b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5" Oct 30 20:41:01.123421 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.123385 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5\" (UID: \"f8cb80d6-3ab3-4cee-90df-745aa3f33e1b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5" Oct 30 20:41:01.123421 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.123423 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5\" (UID: \"f8cb80d6-3ab3-4cee-90df-745aa3f33e1b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5" Oct 30 20:41:01.123812 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.123796 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5\" (UID: \"f8cb80d6-3ab3-4cee-90df-745aa3f33e1b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5" Oct 30 20:41:01.123885 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.123834 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5\" (UID: \"f8cb80d6-3ab3-4cee-90df-745aa3f33e1b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5" Oct 30 20:41:01.132698 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.132679 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g24f\" (UniqueName: \"kubernetes.io/projected/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-kube-api-access-8g24f\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5\" (UID: \"f8cb80d6-3ab3-4cee-90df-745aa3f33e1b\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5" Oct 30 20:41:01.169293 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.169261 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5" Oct 30 20:41:01.266213 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.266184 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc"] Oct 30 20:41:01.269263 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.269245 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.276668 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.276652 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 20:41:01.276957 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.276938 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 20:41:01.277004 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.276944 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 20:41:01.289520 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.289504 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 20:41:01.289594 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.289507 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 20:41:01.289847 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.289836 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 20:41:01.290832 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.290821 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gjgsg\"" Oct 30 20:41:01.305165 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.305141 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc"] Oct 30 20:41:01.426037 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.426012 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.426191 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.426046 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gctlf\" (UniqueName: \"kubernetes.io/projected/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-kube-api-access-gctlf\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.426191 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.426067 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.426191 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.426086 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.426191 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.426157 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.426354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.426223 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.426354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.426321 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.436820 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.436799 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5"] Oct 30 20:41:01.439462 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:41:01.439439 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8cb80d6_3ab3_4cee_90df_745aa3f33e1b.slice/crio-f70719ef76cfbfff20c43b70b9a0ba2f7576c21c1b7e9ab58c3687aa55d92474 WatchSource:0}: Error finding container f70719ef76cfbfff20c43b70b9a0ba2f7576c21c1b7e9ab58c3687aa55d92474: Status 404 returned error can't find the container with id f70719ef76cfbfff20c43b70b9a0ba2f7576c21c1b7e9ab58c3687aa55d92474 Oct 30 20:41:01.498531 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.498502 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5" event={"ID":"f8cb80d6-3ab3-4cee-90df-745aa3f33e1b","Type":"ContainerStarted","Data":"f70719ef76cfbfff20c43b70b9a0ba2f7576c21c1b7e9ab58c3687aa55d92474"} Oct 30 20:41:01.527598 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.527576 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.527718 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.527604 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gctlf\" (UniqueName: \"kubernetes.io/projected/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-kube-api-access-gctlf\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.527718 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.527626 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.527852 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.527716 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.527852 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.527743 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.527852 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.527788 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.527852 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.527834 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.528262 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.528237 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.530161 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.530136 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.530261 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.530213 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.530327 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.530309 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.530438 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.530419 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.537325 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.537300 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gctlf\" (UniqueName: \"kubernetes.io/projected/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-kube-api-access-gctlf\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.537412 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.537338 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jqmqc\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.580339 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.580310 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:01.851425 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:01.851355 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc"] Oct 30 20:41:01.854396 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:41:01.854372 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed2dc2e_2e21_4402_ac8f_0101c97cfc89.slice/crio-9abcc7e7dd29743fd39af09be986195789bb96383680983b44d505ff4dad5ae3 WatchSource:0}: Error finding container 9abcc7e7dd29743fd39af09be986195789bb96383680983b44d505ff4dad5ae3: Status 404 returned error can't find the container with id 9abcc7e7dd29743fd39af09be986195789bb96383680983b44d505ff4dad5ae3 Oct 30 20:41:02.503605 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:02.503565 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" event={"ID":"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89","Type":"ContainerStarted","Data":"9abcc7e7dd29743fd39af09be986195789bb96383680983b44d505ff4dad5ae3"} Oct 30 20:41:05.026088 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:05.026048 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:41:05.026480 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:05.026131 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:41:05.521284 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:05.521245 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" event={"ID":"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89","Type":"ContainerStarted","Data":"ab72f32f5265a199a346afafd6fd9f95461a88786848b515589f4f672b6ae2e1"} Oct 30 20:41:05.521491 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:05.521412 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:05.543916 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:05.543852 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" podStartSLOduration=1.375059022 podStartE2EDuration="4.543834314s" totalImagesPullingTime="3.168775292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:41:05.542135837 +0000 UTC m=+170.722032346" watchObservedRunningTime="2026-10-30 20:41:05.543834314 +0000 UTC m=+170.723730821" Oct 30 20:41:06.527175 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:06.527138 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:41:07.529984 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:07.529894 4848 generic.go:356] "Generic (PLEG): container finished" podID="f8cb80d6-3ab3-4cee-90df-745aa3f33e1b" containerID="80fc8a3e05aa949c1347cb718a68b8270c0c59eb8ed0162f9aa525af095c0056" exitCode=0 Oct 30 20:41:07.530448 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:07.529985 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5" event={"ID":"f8cb80d6-3ab3-4cee-90df-745aa3f33e1b","Type":"ContainerDied","Data":"80fc8a3e05aa949c1347cb718a68b8270c0c59eb8ed0162f9aa525af095c0056"} Oct 30 20:41:08.535426 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:08.535391 4848 generic.go:356] "Generic (PLEG): container finished" podID="f8cb80d6-3ab3-4cee-90df-745aa3f33e1b" containerID="15b7726cdf8aff291b764f58a595eaf81337a77bef83425d6a056db878ec0cca" exitCode=0 Oct 30 20:41:08.535819 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:08.535451 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5" event={"ID":"f8cb80d6-3ab3-4cee-90df-745aa3f33e1b","Type":"ContainerDied","Data":"15b7726cdf8aff291b764f58a595eaf81337a77bef83425d6a056db878ec0cca"} Oct 30 20:41:14.562292 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:14.562260 4848 generic.go:356] "Generic (PLEG): container finished" podID="f8cb80d6-3ab3-4cee-90df-745aa3f33e1b" containerID="a82a9f11653daaeaf28db64dd301709285200b460aeeb3a2642da0b38229c66a" exitCode=0 Oct 30 20:41:14.562292 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:14.562297 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5" event={"ID":"f8cb80d6-3ab3-4cee-90df-745aa3f33e1b","Type":"ContainerDied","Data":"a82a9f11653daaeaf28db64dd301709285200b460aeeb3a2642da0b38229c66a"} Oct 30 20:41:15.691526 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:15.691493 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5" Oct 30 20:41:15.737192 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:15.737155 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-util\" (UniqueName: \"kubernetes.io/empty-dir/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-util\") pod \"f8cb80d6-3ab3-4cee-90df-745aa3f33e1b\" (UID: \"f8cb80d6-3ab3-4cee-90df-745aa3f33e1b\") " Oct 30 20:41:15.737362 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:15.737205 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-bundle\") pod \"f8cb80d6-3ab3-4cee-90df-745aa3f33e1b\" (UID: \"f8cb80d6-3ab3-4cee-90df-745aa3f33e1b\") " Oct 30 20:41:15.737362 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:15.737248 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-kube-api-access-8g24f\" (UniqueName: \"kubernetes.io/projected/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-kube-api-access-8g24f\") pod \"f8cb80d6-3ab3-4cee-90df-745aa3f33e1b\" (UID: \"f8cb80d6-3ab3-4cee-90df-745aa3f33e1b\") " Oct 30 20:41:15.737786 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:15.737741 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-bundle" pod "f8cb80d6-3ab3-4cee-90df-745aa3f33e1b" (UID: "f8cb80d6-3ab3-4cee-90df-745aa3f33e1b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:41:15.739367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:15.739344 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-kube-api-access-8g24f" pod "f8cb80d6-3ab3-4cee-90df-745aa3f33e1b" (UID: "f8cb80d6-3ab3-4cee-90df-745aa3f33e1b"). InnerVolumeSpecName "kube-api-access-8g24f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:41:15.744392 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:15.744366 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-util" pod "f8cb80d6-3ab3-4cee-90df-745aa3f33e1b" (UID: "f8cb80d6-3ab3-4cee-90df-745aa3f33e1b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:41:15.838414 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:15.838321 4848 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-util\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:15.838414 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:15.838365 4848 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-bundle\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:15.838414 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:15.838375 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g24f\" (UniqueName: \"kubernetes.io/projected/f8cb80d6-3ab3-4cee-90df-745aa3f33e1b-kube-api-access-8g24f\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:41:16.571028 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:16.570987 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5" event={"ID":"f8cb80d6-3ab3-4cee-90df-745aa3f33e1b","Type":"ContainerDied","Data":"f70719ef76cfbfff20c43b70b9a0ba2f7576c21c1b7e9ab58c3687aa55d92474"} Oct 30 20:41:16.571028 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:16.571008 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hgbf5" Oct 30 20:41:16.571028 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:16.571033 4848 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f70719ef76cfbfff20c43b70b9a0ba2f7576c21c1b7e9ab58c3687aa55d92474" Oct 30 20:41:22.508010 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:41:22.507976 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:41:52.509687 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:41:52.509605 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:41:55.350852 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.350820 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh"] Oct 30 20:41:55.351310 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.351093 4848 cpu_manager.go:414] "containerMap: removing container" podUID="f8cb80d6-3ab3-4cee-90df-745aa3f33e1b" containerName="util" Oct 30 20:41:55.351310 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.351108 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8cb80d6-3ab3-4cee-90df-745aa3f33e1b" containerName="util" Oct 30 20:41:55.351310 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.351120 4848 cpu_manager.go:414] "containerMap: removing container" podUID="f8cb80d6-3ab3-4cee-90df-745aa3f33e1b" containerName="pull" Oct 30 20:41:55.351310 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.351128 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8cb80d6-3ab3-4cee-90df-745aa3f33e1b" containerName="pull" Oct 30 20:41:55.351310 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.351138 4848 cpu_manager.go:414] "containerMap: removing container" podUID="f8cb80d6-3ab3-4cee-90df-745aa3f33e1b" containerName="extract" Oct 30 20:41:55.351310 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.351145 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8cb80d6-3ab3-4cee-90df-745aa3f33e1b" containerName="extract" Oct 30 20:41:55.351310 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.351225 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8cb80d6-3ab3-4cee-90df-745aa3f33e1b" containerName="extract" Oct 30 20:41:55.353132 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.353118 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:41:55.387492 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.387461 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh"] Oct 30 20:41:55.411649 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.411618 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/00202fb8-a18e-4edd-9199-e136a6d35cd6-metrics-cert\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-qbdqh\" (UID: \"00202fb8-a18e-4edd-9199-e136a6d35cd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:41:55.411826 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.411665 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00202fb8-a18e-4edd-9199-e136a6d35cd6-tmp\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-qbdqh\" (UID: \"00202fb8-a18e-4edd-9199-e136a6d35cd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:41:55.411826 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.411705 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/00202fb8-a18e-4edd-9199-e136a6d35cd6-webhook-cert\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-qbdqh\" (UID: \"00202fb8-a18e-4edd-9199-e136a6d35cd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:41:55.411951 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.411886 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d77jm\" (UniqueName: \"kubernetes.io/projected/00202fb8-a18e-4edd-9199-e136a6d35cd6-kube-api-access-d77jm\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-qbdqh\" (UID: \"00202fb8-a18e-4edd-9199-e136a6d35cd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:41:55.411988 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.411960 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/00202fb8-a18e-4edd-9199-e136a6d35cd6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-qbdqh\" (UID: \"00202fb8-a18e-4edd-9199-e136a6d35cd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:41:55.412022 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.411997 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/00202fb8-a18e-4edd-9199-e136a6d35cd6-manifests\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-qbdqh\" (UID: \"00202fb8-a18e-4edd-9199-e136a6d35cd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:41:55.413420 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.413400 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 20:41:55.413657 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.413638 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 20:41:55.413951 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.413937 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-x2mvh\"" Oct 30 20:41:55.414124 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.414110 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 20:41:55.414312 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.414297 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 20:41:55.414459 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.414447 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 20:41:55.513143 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.513111 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/00202fb8-a18e-4edd-9199-e136a6d35cd6-webhook-cert\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-qbdqh\" (UID: \"00202fb8-a18e-4edd-9199-e136a6d35cd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:41:55.513143 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.513147 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d77jm\" (UniqueName: \"kubernetes.io/projected/00202fb8-a18e-4edd-9199-e136a6d35cd6-kube-api-access-d77jm\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-qbdqh\" (UID: \"00202fb8-a18e-4edd-9199-e136a6d35cd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:41:55.513390 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.513172 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/00202fb8-a18e-4edd-9199-e136a6d35cd6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-qbdqh\" (UID: \"00202fb8-a18e-4edd-9199-e136a6d35cd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:41:55.513390 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.513294 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/00202fb8-a18e-4edd-9199-e136a6d35cd6-manifests\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-qbdqh\" (UID: \"00202fb8-a18e-4edd-9199-e136a6d35cd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:41:55.513390 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.513330 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/00202fb8-a18e-4edd-9199-e136a6d35cd6-metrics-cert\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-qbdqh\" (UID: \"00202fb8-a18e-4edd-9199-e136a6d35cd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:41:55.513390 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.513381 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00202fb8-a18e-4edd-9199-e136a6d35cd6-tmp\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-qbdqh\" (UID: \"00202fb8-a18e-4edd-9199-e136a6d35cd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:41:55.513721 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.513688 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/00202fb8-a18e-4edd-9199-e136a6d35cd6-manifests\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-qbdqh\" (UID: \"00202fb8-a18e-4edd-9199-e136a6d35cd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:41:55.515578 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.515551 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/00202fb8-a18e-4edd-9199-e136a6d35cd6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-qbdqh\" (UID: \"00202fb8-a18e-4edd-9199-e136a6d35cd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:41:55.515692 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.515674 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00202fb8-a18e-4edd-9199-e136a6d35cd6-tmp\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-qbdqh\" (UID: \"00202fb8-a18e-4edd-9199-e136a6d35cd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:41:55.515733 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.515675 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/00202fb8-a18e-4edd-9199-e136a6d35cd6-metrics-cert\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-qbdqh\" (UID: \"00202fb8-a18e-4edd-9199-e136a6d35cd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:41:55.515889 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.515875 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/00202fb8-a18e-4edd-9199-e136a6d35cd6-webhook-cert\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-qbdqh\" (UID: \"00202fb8-a18e-4edd-9199-e136a6d35cd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:41:55.538468 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.538438 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d77jm\" (UniqueName: \"kubernetes.io/projected/00202fb8-a18e-4edd-9199-e136a6d35cd6-kube-api-access-d77jm\") pod \"opendatahub-operator-controller-manager-597bf9dc6d-qbdqh\" (UID: \"00202fb8-a18e-4edd-9199-e136a6d35cd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:41:55.665513 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.665442 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:41:55.968487 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:55.968447 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh"] Oct 30 20:41:55.973370 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:41:55.973334 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00202fb8_a18e_4edd_9199_e136a6d35cd6.slice/crio-296c196b8fbc3f887885ad46886934bdf09c343cf859e88f342b2d2dd7d23d2e WatchSource:0}: Error finding container 296c196b8fbc3f887885ad46886934bdf09c343cf859e88f342b2d2dd7d23d2e: Status 404 returned error can't find the container with id 296c196b8fbc3f887885ad46886934bdf09c343cf859e88f342b2d2dd7d23d2e Oct 30 20:41:56.702478 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:56.702408 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" event={"ID":"00202fb8-a18e-4edd-9199-e136a6d35cd6","Type":"ContainerStarted","Data":"296c196b8fbc3f887885ad46886934bdf09c343cf859e88f342b2d2dd7d23d2e"} Oct 30 20:41:59.717519 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:59.717488 4848 generic.go:356] "Generic (PLEG): container finished" podID="00202fb8-a18e-4edd-9199-e136a6d35cd6" containerID="4638c430ed9f18eb9ca7974c3c0c39ac8b9e7edd3e1adc7b47c1c609748bb94b" exitCode=0 Oct 30 20:41:59.717919 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:41:59.717545 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" event={"ID":"00202fb8-a18e-4edd-9199-e136a6d35cd6","Type":"ContainerDied","Data":"4638c430ed9f18eb9ca7974c3c0c39ac8b9e7edd3e1adc7b47c1c609748bb94b"} Oct 30 20:42:00.722245 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:00.722207 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" event={"ID":"00202fb8-a18e-4edd-9199-e136a6d35cd6","Type":"ContainerStarted","Data":"12f18e351e5a4c4d1e66c146e5cafcae2951002b833fe1a2393b1a41f1780f4e"} Oct 30 20:42:00.722644 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:00.722344 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:42:00.757589 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:00.757532 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" podStartSLOduration=2.06291596 podStartE2EDuration="5.757518368s" totalImagesPullingTime="3.273113615s" totalInitContainerRuntime="421.488793ms" isStatefulPod=false podCreationTimestamp="2026-10-30 20:41:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:00.755464019 +0000 UTC m=+225.935360525" watchObservedRunningTime="2026-10-30 20:42:00.757518368 +0000 UTC m=+225.937414874" Oct 30 20:42:11.727785 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:11.727737 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-597bf9dc6d-qbdqh" Oct 30 20:42:17.771873 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:17.771834 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-tpc9x"] Oct 30 20:42:17.778787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:17.778749 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tpc9x" Oct 30 20:42:17.786124 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:17.786099 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tpc9x"] Oct 30 20:42:17.788250 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:17.788234 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 20:42:17.788315 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:17.788280 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 20:42:17.807058 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:17.807028 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-k8fzw\"" Oct 30 20:42:17.885311 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:17.885267 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd844e2c-555e-4e4b-9d36-1db43726ab4f-bound-sa-token\") pod \"cert-manager-597b95cb84-tpc9x\" (UID: \"bd844e2c-555e-4e4b-9d36-1db43726ab4f\") " pod="cert-manager/cert-manager-597b95cb84-tpc9x" Oct 30 20:42:17.885311 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:17.885314 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg9pr\" (UniqueName: \"kubernetes.io/projected/bd844e2c-555e-4e4b-9d36-1db43726ab4f-kube-api-access-zg9pr\") pod \"cert-manager-597b95cb84-tpc9x\" (UID: \"bd844e2c-555e-4e4b-9d36-1db43726ab4f\") " pod="cert-manager/cert-manager-597b95cb84-tpc9x" Oct 30 20:42:17.986129 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:17.986094 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd844e2c-555e-4e4b-9d36-1db43726ab4f-bound-sa-token\") pod \"cert-manager-597b95cb84-tpc9x\" (UID: \"bd844e2c-555e-4e4b-9d36-1db43726ab4f\") " pod="cert-manager/cert-manager-597b95cb84-tpc9x" Oct 30 20:42:17.986308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:17.986137 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg9pr\" (UniqueName: \"kubernetes.io/projected/bd844e2c-555e-4e4b-9d36-1db43726ab4f-kube-api-access-zg9pr\") pod \"cert-manager-597b95cb84-tpc9x\" (UID: \"bd844e2c-555e-4e4b-9d36-1db43726ab4f\") " pod="cert-manager/cert-manager-597b95cb84-tpc9x" Oct 30 20:42:17.994915 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:17.994890 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bd844e2c-555e-4e4b-9d36-1db43726ab4f-bound-sa-token\") pod \"cert-manager-597b95cb84-tpc9x\" (UID: \"bd844e2c-555e-4e4b-9d36-1db43726ab4f\") " pod="cert-manager/cert-manager-597b95cb84-tpc9x" Oct 30 20:42:17.995039 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:17.994979 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg9pr\" (UniqueName: \"kubernetes.io/projected/bd844e2c-555e-4e4b-9d36-1db43726ab4f-kube-api-access-zg9pr\") pod \"cert-manager-597b95cb84-tpc9x\" (UID: \"bd844e2c-555e-4e4b-9d36-1db43726ab4f\") " pod="cert-manager/cert-manager-597b95cb84-tpc9x" Oct 30 20:42:18.111027 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:18.110920 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tpc9x" Oct 30 20:42:18.379610 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:18.379531 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tpc9x"] Oct 30 20:42:18.382389 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:42:18.382362 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd844e2c_555e_4e4b_9d36_1db43726ab4f.slice/crio-2ab87791bb7b64147c483bacd6cc8bbe8dc95c65986cac77fbb36fcaef65dd38 WatchSource:0}: Error finding container 2ab87791bb7b64147c483bacd6cc8bbe8dc95c65986cac77fbb36fcaef65dd38: Status 404 returned error can't find the container with id 2ab87791bb7b64147c483bacd6cc8bbe8dc95c65986cac77fbb36fcaef65dd38 Oct 30 20:42:18.782620 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:18.782585 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tpc9x" event={"ID":"bd844e2c-555e-4e4b-9d36-1db43726ab4f","Type":"ContainerStarted","Data":"2ab87791bb7b64147c483bacd6cc8bbe8dc95c65986cac77fbb36fcaef65dd38"} Oct 30 20:42:22.507757 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:42:22.507718 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:42:22.754063 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:22.754026 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-64cbf78866-9kpj2"] Oct 30 20:42:22.757191 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:22.757174 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64cbf78866-9kpj2" Oct 30 20:42:22.775903 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:22.775860 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64cbf78866-9kpj2"] Oct 30 20:42:22.776934 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:22.776799 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 20:42:22.776934 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:22.776816 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 20:42:22.776934 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:22.776907 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6zrmq\"" Oct 30 20:42:22.799209 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:22.799179 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tpc9x" event={"ID":"bd844e2c-555e-4e4b-9d36-1db43726ab4f","Type":"ContainerStarted","Data":"e8f711176ac84fab43ace265df70cc8db2c9aebfe617963fdac2cd354fa4d84a"} Oct 30 20:42:22.826268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:22.826241 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c-tls-certs\") pod \"kube-auth-proxy-64cbf78866-9kpj2\" (UID: \"c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c\") " pod="openshift-ingress/kube-auth-proxy-64cbf78866-9kpj2" Oct 30 20:42:22.826387 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:22.826297 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n7wk\" (UniqueName: \"kubernetes.io/projected/c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c-kube-api-access-6n7wk\") pod \"kube-auth-proxy-64cbf78866-9kpj2\" (UID: \"c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c\") " pod="openshift-ingress/kube-auth-proxy-64cbf78866-9kpj2" Oct 30 20:42:22.826387 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:22.826320 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c-tmp\") pod \"kube-auth-proxy-64cbf78866-9kpj2\" (UID: \"c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c\") " pod="openshift-ingress/kube-auth-proxy-64cbf78866-9kpj2" Oct 30 20:42:22.849717 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:22.849669 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-tpc9x" podStartSLOduration=2.121257721 podStartE2EDuration="5.849657145s" totalImagesPullingTime="3.728399424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:22.84938027 +0000 UTC m=+248.029276779" watchObservedRunningTime="2026-10-30 20:42:22.849657145 +0000 UTC m=+248.029553654" Oct 30 20:42:22.927059 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:22.927017 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c-tmp\") pod \"kube-auth-proxy-64cbf78866-9kpj2\" (UID: \"c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c\") " pod="openshift-ingress/kube-auth-proxy-64cbf78866-9kpj2" Oct 30 20:42:22.927243 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:22.927119 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c-tls-certs\") pod \"kube-auth-proxy-64cbf78866-9kpj2\" (UID: \"c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c\") " pod="openshift-ingress/kube-auth-proxy-64cbf78866-9kpj2" Oct 30 20:42:22.927243 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:22.927234 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n7wk\" (UniqueName: \"kubernetes.io/projected/c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c-kube-api-access-6n7wk\") pod \"kube-auth-proxy-64cbf78866-9kpj2\" (UID: \"c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c\") " pod="openshift-ingress/kube-auth-proxy-64cbf78866-9kpj2" Oct 30 20:42:22.929567 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:22.929534 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c-tmp\") pod \"kube-auth-proxy-64cbf78866-9kpj2\" (UID: \"c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c\") " pod="openshift-ingress/kube-auth-proxy-64cbf78866-9kpj2" Oct 30 20:42:22.931389 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:22.931363 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c-tls-certs\") pod \"kube-auth-proxy-64cbf78866-9kpj2\" (UID: \"c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c\") " pod="openshift-ingress/kube-auth-proxy-64cbf78866-9kpj2" Oct 30 20:42:22.938871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:22.938850 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n7wk\" (UniqueName: \"kubernetes.io/projected/c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c-kube-api-access-6n7wk\") pod \"kube-auth-proxy-64cbf78866-9kpj2\" (UID: \"c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c\") " pod="openshift-ingress/kube-auth-proxy-64cbf78866-9kpj2" Oct 30 20:42:23.068283 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:23.068206 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64cbf78866-9kpj2" Oct 30 20:42:23.348489 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:23.348457 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64cbf78866-9kpj2"] Oct 30 20:42:23.348660 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:42:23.348510 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ac8ee6_0f8b_40db_b97c_cd9ecf694f6c.slice/crio-98644e7ebd35ae782e615aea416b02a7a35693075561b0e6755d48db346f02bf WatchSource:0}: Error finding container 98644e7ebd35ae782e615aea416b02a7a35693075561b0e6755d48db346f02bf: Status 404 returned error can't find the container with id 98644e7ebd35ae782e615aea416b02a7a35693075561b0e6755d48db346f02bf Oct 30 20:42:23.803304 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:23.803266 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64cbf78866-9kpj2" event={"ID":"c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c","Type":"ContainerStarted","Data":"98644e7ebd35ae782e615aea416b02a7a35693075561b0e6755d48db346f02bf"} Oct 30 20:42:25.816135 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:25.816098 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64cbf78866-9kpj2" event={"ID":"c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c","Type":"ContainerStarted","Data":"e4c69e46dc7dc953aaf35557aec4bf12e1aa3b031dcf3c509204081344a4b0b5"} Oct 30 20:42:25.838481 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:42:25.838428 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-64cbf78866-9kpj2" podStartSLOduration=1.481665595 podStartE2EDuration="3.838411023s" totalImagesPullingTime="2.356745428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:42:25.837314549 +0000 UTC m=+251.017211055" watchObservedRunningTime="2026-10-30 20:42:25.838411023 +0000 UTC m=+251.018307529" Oct 30 20:42:52.508327 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:42:52.508290 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:43:05.953374 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:05.953342 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jpzw8"] Oct 30 20:43:05.961799 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:05.961756 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jpzw8" Oct 30 20:43:05.965301 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:05.965278 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jpzw8"] Oct 30 20:43:05.969750 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:05.969718 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bdghc\"" Oct 30 20:43:05.969868 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:05.969801 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 20:43:05.969868 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:05.969823 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 20:43:05.970067 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:05.970049 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 20:43:06.035411 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:06.035382 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d2f37d2-e737-4f6b-be8a-c36c5bead2b6-config-file\") pod \"limitador-limitador-74dfcfb894-jpzw8\" (UID: \"7d2f37d2-e737-4f6b-be8a-c36c5bead2b6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jpzw8" Oct 30 20:43:06.035585 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:06.035478 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctb6d\" (UniqueName: \"kubernetes.io/projected/7d2f37d2-e737-4f6b-be8a-c36c5bead2b6-kube-api-access-ctb6d\") pod \"limitador-limitador-74dfcfb894-jpzw8\" (UID: \"7d2f37d2-e737-4f6b-be8a-c36c5bead2b6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jpzw8" Oct 30 20:43:06.057126 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:06.057096 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jpzw8"] Oct 30 20:43:06.136801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:06.136746 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d2f37d2-e737-4f6b-be8a-c36c5bead2b6-config-file\") pod \"limitador-limitador-74dfcfb894-jpzw8\" (UID: \"7d2f37d2-e737-4f6b-be8a-c36c5bead2b6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jpzw8" Oct 30 20:43:06.136947 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:06.136818 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctb6d\" (UniqueName: \"kubernetes.io/projected/7d2f37d2-e737-4f6b-be8a-c36c5bead2b6-kube-api-access-ctb6d\") pod \"limitador-limitador-74dfcfb894-jpzw8\" (UID: \"7d2f37d2-e737-4f6b-be8a-c36c5bead2b6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jpzw8" Oct 30 20:43:06.137400 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:06.137378 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d2f37d2-e737-4f6b-be8a-c36c5bead2b6-config-file\") pod \"limitador-limitador-74dfcfb894-jpzw8\" (UID: \"7d2f37d2-e737-4f6b-be8a-c36c5bead2b6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jpzw8" Oct 30 20:43:06.145902 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:06.145879 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctb6d\" (UniqueName: \"kubernetes.io/projected/7d2f37d2-e737-4f6b-be8a-c36c5bead2b6-kube-api-access-ctb6d\") pod \"limitador-limitador-74dfcfb894-jpzw8\" (UID: \"7d2f37d2-e737-4f6b-be8a-c36c5bead2b6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jpzw8" Oct 30 20:43:06.275188 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:06.275105 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jpzw8" Oct 30 20:43:06.543817 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:06.543794 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jpzw8"] Oct 30 20:43:06.546431 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:43:06.546404 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2f37d2_e737_4f6b_be8a_c36c5bead2b6.slice/crio-d91efc7248e440bccd5a8140448cb4650da4d9bf6c44e5823482dda28cb2a4f6 WatchSource:0}: Error finding container d91efc7248e440bccd5a8140448cb4650da4d9bf6c44e5823482dda28cb2a4f6: Status 404 returned error can't find the container with id d91efc7248e440bccd5a8140448cb4650da4d9bf6c44e5823482dda28cb2a4f6 Oct 30 20:43:06.916463 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:06.916383 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-27ckg"] Oct 30 20:43:06.920498 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:06.920481 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-27ckg" Oct 30 20:43:06.931025 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:06.931000 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xhj5t\"" Oct 30 20:43:06.931341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:06.931322 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-27ckg"] Oct 30 20:43:06.943609 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:06.943581 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-829vb\" (UniqueName: \"kubernetes.io/projected/859796d1-55c4-49b4-87b3-8280fca7564a-kube-api-access-829vb\") pod \"authorino-57f7f8659f-27ckg\" (UID: \"859796d1-55c4-49b4-87b3-8280fca7564a\") " pod="kuadrant-system/authorino-57f7f8659f-27ckg" Oct 30 20:43:06.960338 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:06.960307 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jpzw8" event={"ID":"7d2f37d2-e737-4f6b-be8a-c36c5bead2b6","Type":"ContainerStarted","Data":"d91efc7248e440bccd5a8140448cb4650da4d9bf6c44e5823482dda28cb2a4f6"} Oct 30 20:43:07.044387 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:07.044352 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-829vb\" (UniqueName: \"kubernetes.io/projected/859796d1-55c4-49b4-87b3-8280fca7564a-kube-api-access-829vb\") pod \"authorino-57f7f8659f-27ckg\" (UID: \"859796d1-55c4-49b4-87b3-8280fca7564a\") " pod="kuadrant-system/authorino-57f7f8659f-27ckg" Oct 30 20:43:07.052928 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:07.052907 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-829vb\" (UniqueName: \"kubernetes.io/projected/859796d1-55c4-49b4-87b3-8280fca7564a-kube-api-access-829vb\") pod \"authorino-57f7f8659f-27ckg\" (UID: \"859796d1-55c4-49b4-87b3-8280fca7564a\") " pod="kuadrant-system/authorino-57f7f8659f-27ckg" Oct 30 20:43:07.100073 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:07.100045 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zpnjg"] Oct 30 20:43:07.103080 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:07.103062 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zpnjg" Oct 30 20:43:07.110312 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:07.110288 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zpnjg"] Oct 30 20:43:07.145538 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:07.145512 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74lct\" (UniqueName: \"kubernetes.io/projected/087322b1-be2a-4aea-a31d-3e5d2fcd2340-kube-api-access-74lct\") pod \"authorino-56c6c9ddd5-zpnjg\" (UID: \"087322b1-be2a-4aea-a31d-3e5d2fcd2340\") " pod="kuadrant-system/authorino-56c6c9ddd5-zpnjg" Oct 30 20:43:07.234296 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:07.234268 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-27ckg" Oct 30 20:43:07.246295 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:07.246258 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74lct\" (UniqueName: \"kubernetes.io/projected/087322b1-be2a-4aea-a31d-3e5d2fcd2340-kube-api-access-74lct\") pod \"authorino-56c6c9ddd5-zpnjg\" (UID: \"087322b1-be2a-4aea-a31d-3e5d2fcd2340\") " pod="kuadrant-system/authorino-56c6c9ddd5-zpnjg" Oct 30 20:43:07.255813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:07.255781 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74lct\" (UniqueName: \"kubernetes.io/projected/087322b1-be2a-4aea-a31d-3e5d2fcd2340-kube-api-access-74lct\") pod \"authorino-56c6c9ddd5-zpnjg\" (UID: \"087322b1-be2a-4aea-a31d-3e5d2fcd2340\") " pod="kuadrant-system/authorino-56c6c9ddd5-zpnjg" Oct 30 20:43:07.415182 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:07.415147 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zpnjg" Oct 30 20:43:07.502630 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:07.502563 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-27ckg"] Oct 30 20:43:07.505493 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:43:07.505468 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod859796d1_55c4_49b4_87b3_8280fca7564a.slice/crio-88352cb05a57cddfcd400928ad726ae084a26f3e503f1e437f3cd369c664c45f WatchSource:0}: Error finding container 88352cb05a57cddfcd400928ad726ae084a26f3e503f1e437f3cd369c664c45f: Status 404 returned error can't find the container with id 88352cb05a57cddfcd400928ad726ae084a26f3e503f1e437f3cd369c664c45f Oct 30 20:43:07.685735 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:07.685674 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zpnjg"] Oct 30 20:43:07.689599 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:43:07.689574 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod087322b1_be2a_4aea_a31d_3e5d2fcd2340.slice/crio-647e9bef18316299fcffdd02a85b73293f0a81bb81484347ee94986ff0ba60b4 WatchSource:0}: Error finding container 647e9bef18316299fcffdd02a85b73293f0a81bb81484347ee94986ff0ba60b4: Status 404 returned error can't find the container with id 647e9bef18316299fcffdd02a85b73293f0a81bb81484347ee94986ff0ba60b4 Oct 30 20:43:07.965729 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:07.965686 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zpnjg" event={"ID":"087322b1-be2a-4aea-a31d-3e5d2fcd2340","Type":"ContainerStarted","Data":"647e9bef18316299fcffdd02a85b73293f0a81bb81484347ee94986ff0ba60b4"} Oct 30 20:43:07.966980 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:07.966947 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-27ckg" event={"ID":"859796d1-55c4-49b4-87b3-8280fca7564a","Type":"ContainerStarted","Data":"88352cb05a57cddfcd400928ad726ae084a26f3e503f1e437f3cd369c664c45f"} Oct 30 20:43:09.977474 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:09.977434 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jpzw8" event={"ID":"7d2f37d2-e737-4f6b-be8a-c36c5bead2b6","Type":"ContainerStarted","Data":"96a3044e8c159d511c41fd34e9aa33d0d42bdb33a23db06fc85c73d6f94bb664"} Oct 30 20:43:09.977967 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:09.977555 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-jpzw8" Oct 30 20:43:09.996748 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:09.996690 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-jpzw8" podStartSLOduration=1.906790861 podStartE2EDuration="4.996672726s" totalImagesPullingTime="3.089881865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:09.995577588 +0000 UTC m=+295.175474095" watchObservedRunningTime="2026-10-30 20:43:09.996672726 +0000 UTC m=+295.176569235" Oct 30 20:43:10.982276 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:10.982231 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zpnjg" event={"ID":"087322b1-be2a-4aea-a31d-3e5d2fcd2340","Type":"ContainerStarted","Data":"ed26988b2adc7c099315703a87e40f9f1d5adc5e2c1673907486f237d1732197"} Oct 30 20:43:10.983644 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:10.983616 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-27ckg" event={"ID":"859796d1-55c4-49b4-87b3-8280fca7564a","Type":"ContainerStarted","Data":"1a7374484a57c14777f70c3947c4e95ee2e27d22d00d6bbeefdd1e6d0ef9ea64"} Oct 30 20:43:11.005990 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:11.005946 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-zpnjg" podStartSLOduration=1.367468341 podStartE2EDuration="4.005933177s" totalImagesPullingTime="2.638464836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:11.004923857 +0000 UTC m=+296.184820366" watchObservedRunningTime="2026-10-30 20:43:11.005933177 +0000 UTC m=+296.185829683" Oct 30 20:43:11.040216 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:11.040151 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-27ckg" podStartSLOduration=2.224897732 podStartE2EDuration="5.040134897s" totalImagesPullingTime="2.815237165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:11.040039159 +0000 UTC m=+296.219935664" watchObservedRunningTime="2026-10-30 20:43:11.040134897 +0000 UTC m=+296.220031404" Oct 30 20:43:11.058784 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:11.058743 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-27ckg"] Oct 30 20:43:12.991271 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:12.991235 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-27ckg" podUID="859796d1-55c4-49b4-87b3-8280fca7564a" containerName="authorino" containerID="cri-o://1a7374484a57c14777f70c3947c4e95ee2e27d22d00d6bbeefdd1e6d0ef9ea64" gracePeriod=30 Oct 30 20:43:13.222225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:13.222200 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-27ckg" Oct 30 20:43:13.391106 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:13.391022 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/859796d1-55c4-49b4-87b3-8280fca7564a-kube-api-access-829vb\" (UniqueName: \"kubernetes.io/projected/859796d1-55c4-49b4-87b3-8280fca7564a-kube-api-access-829vb\") pod \"859796d1-55c4-49b4-87b3-8280fca7564a\" (UID: \"859796d1-55c4-49b4-87b3-8280fca7564a\") " Oct 30 20:43:13.393192 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:13.393165 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859796d1-55c4-49b4-87b3-8280fca7564a-kube-api-access-829vb" pod "859796d1-55c4-49b4-87b3-8280fca7564a" (UID: "859796d1-55c4-49b4-87b3-8280fca7564a"). InnerVolumeSpecName "kube-api-access-829vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:43:13.491999 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:13.491968 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-829vb\" (UniqueName: \"kubernetes.io/projected/859796d1-55c4-49b4-87b3-8280fca7564a-kube-api-access-829vb\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:43:13.995865 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:13.995828 4848 generic.go:356] "Generic (PLEG): container finished" podID="859796d1-55c4-49b4-87b3-8280fca7564a" containerID="1a7374484a57c14777f70c3947c4e95ee2e27d22d00d6bbeefdd1e6d0ef9ea64" exitCode=0 Oct 30 20:43:13.996296 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:13.995911 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-27ckg" Oct 30 20:43:13.996296 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:13.995944 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-27ckg" event={"ID":"859796d1-55c4-49b4-87b3-8280fca7564a","Type":"ContainerDied","Data":"1a7374484a57c14777f70c3947c4e95ee2e27d22d00d6bbeefdd1e6d0ef9ea64"} Oct 30 20:43:13.996296 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:13.995981 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-27ckg" event={"ID":"859796d1-55c4-49b4-87b3-8280fca7564a","Type":"ContainerDied","Data":"88352cb05a57cddfcd400928ad726ae084a26f3e503f1e437f3cd369c664c45f"} Oct 30 20:43:13.996296 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:13.996007 4848 scope.go:122] "RemoveContainer" containerID="1a7374484a57c14777f70c3947c4e95ee2e27d22d00d6bbeefdd1e6d0ef9ea64" Oct 30 20:43:14.004220 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:14.004206 4848 scope.go:122] "RemoveContainer" containerID="1a7374484a57c14777f70c3947c4e95ee2e27d22d00d6bbeefdd1e6d0ef9ea64" Oct 30 20:43:14.004509 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:43:14.004484 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a7374484a57c14777f70c3947c4e95ee2e27d22d00d6bbeefdd1e6d0ef9ea64\": container with ID starting with 1a7374484a57c14777f70c3947c4e95ee2e27d22d00d6bbeefdd1e6d0ef9ea64 not found: ID does not exist" containerID="1a7374484a57c14777f70c3947c4e95ee2e27d22d00d6bbeefdd1e6d0ef9ea64" Oct 30 20:43:14.004602 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:14.004513 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a7374484a57c14777f70c3947c4e95ee2e27d22d00d6bbeefdd1e6d0ef9ea64"} err="failed to get container status \"1a7374484a57c14777f70c3947c4e95ee2e27d22d00d6bbeefdd1e6d0ef9ea64\": rpc error: code = NotFound desc = could not find container \"1a7374484a57c14777f70c3947c4e95ee2e27d22d00d6bbeefdd1e6d0ef9ea64\": container with ID starting with 1a7374484a57c14777f70c3947c4e95ee2e27d22d00d6bbeefdd1e6d0ef9ea64 not found: ID does not exist" Oct 30 20:43:14.020595 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:14.020573 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-27ckg"] Oct 30 20:43:14.025697 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:14.025653 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-27ckg"] Oct 30 20:43:14.929920 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:14.929892 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="859796d1-55c4-49b4-87b3-8280fca7564a" path="/var/lib/kubelet/pods/859796d1-55c4-49b4-87b3-8280fca7564a/volumes" Oct 30 20:43:14.954837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:14.954812 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 20:43:14.955044 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:14.955025 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 20:43:18.842657 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.842626 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-chpzx"] Oct 30 20:43:18.843078 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.842942 4848 cpu_manager.go:414] "containerMap: removing container" podUID="859796d1-55c4-49b4-87b3-8280fca7564a" containerName="authorino" Oct 30 20:43:18.843078 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.842954 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="859796d1-55c4-49b4-87b3-8280fca7564a" containerName="authorino" Oct 30 20:43:18.843078 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.843029 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="859796d1-55c4-49b4-87b3-8280fca7564a" containerName="authorino" Oct 30 20:43:18.847376 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.847357 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-chpzx" Oct 30 20:43:18.858330 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.858313 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 20:43:18.858594 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.858574 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 20:43:18.860435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.860407 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-chpzx"] Oct 30 20:43:18.871933 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.871911 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 20:43:18.871933 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.871911 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-tgdpr\"" Oct 30 20:43:18.894115 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.894088 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-htl5n"] Oct 30 20:43:18.897148 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.897130 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-htl5n" Oct 30 20:43:18.907762 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.907699 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-htl5n"] Oct 30 20:43:18.913406 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.913381 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-btqgb"] Oct 30 20:43:18.916675 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.916655 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-btqgb" Oct 30 20:43:18.932222 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.932198 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-btqgb"] Oct 30 20:43:18.936653 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.936629 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtpc8\" (UniqueName: \"kubernetes.io/projected/30fe2f63-49a7-43f5-9530-9f1cf25d635c-kube-api-access-vtpc8\") pod \"kserve-controller-manager-776cc66d47-chpzx\" (UID: \"30fe2f63-49a7-43f5-9530-9f1cf25d635c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-chpzx" Oct 30 20:43:18.936760 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.936664 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26-metrics-certs\") pod \"odh-model-controller-769875db56-btqgb\" (UID: \"ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26\") " pod="opendatahub/odh-model-controller-769875db56-btqgb" Oct 30 20:43:18.936760 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.936687 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c746478-bdc8-45c9-b5da-39a1fe98f1d1-tls-certs\") pod \"model-serving-api-75bd89fdfc-htl5n\" (UID: \"8c746478-bdc8-45c9-b5da-39a1fe98f1d1\") " pod="opendatahub/model-serving-api-75bd89fdfc-htl5n" Oct 30 20:43:18.936905 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.936880 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26-cert\") pod \"odh-model-controller-769875db56-btqgb\" (UID: \"ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26\") " pod="opendatahub/odh-model-controller-769875db56-btqgb" Oct 30 20:43:18.936970 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.936919 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30fe2f63-49a7-43f5-9530-9f1cf25d635c-cert\") pod \"kserve-controller-manager-776cc66d47-chpzx\" (UID: \"30fe2f63-49a7-43f5-9530-9f1cf25d635c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-chpzx" Oct 30 20:43:18.936970 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.936959 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtqft\" (UniqueName: \"kubernetes.io/projected/ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26-kube-api-access-rtqft\") pod \"odh-model-controller-769875db56-btqgb\" (UID: \"ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26\") " pod="opendatahub/odh-model-controller-769875db56-btqgb" Oct 30 20:43:18.937039 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.937016 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xh67\" (UniqueName: \"kubernetes.io/projected/8c746478-bdc8-45c9-b5da-39a1fe98f1d1-kube-api-access-7xh67\") pod \"model-serving-api-75bd89fdfc-htl5n\" (UID: \"8c746478-bdc8-45c9-b5da-39a1fe98f1d1\") " pod="opendatahub/model-serving-api-75bd89fdfc-htl5n" Oct 30 20:43:18.965613 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.965574 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 20:43:18.965613 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.965589 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 20:43:18.965812 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.965623 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 20:43:18.965812 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.965659 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-h8csl\"" Oct 30 20:43:18.965812 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:18.965686 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-dptfm\"" Oct 30 20:43:19.038134 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.038092 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26-cert\") pod \"odh-model-controller-769875db56-btqgb\" (UID: \"ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26\") " pod="opendatahub/odh-model-controller-769875db56-btqgb" Oct 30 20:43:19.038322 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.038164 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30fe2f63-49a7-43f5-9530-9f1cf25d635c-cert\") pod \"kserve-controller-manager-776cc66d47-chpzx\" (UID: \"30fe2f63-49a7-43f5-9530-9f1cf25d635c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-chpzx" Oct 30 20:43:19.038322 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.038208 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtqft\" (UniqueName: \"kubernetes.io/projected/ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26-kube-api-access-rtqft\") pod \"odh-model-controller-769875db56-btqgb\" (UID: \"ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26\") " pod="opendatahub/odh-model-controller-769875db56-btqgb" Oct 30 20:43:19.038322 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.038246 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xh67\" (UniqueName: \"kubernetes.io/projected/8c746478-bdc8-45c9-b5da-39a1fe98f1d1-kube-api-access-7xh67\") pod \"model-serving-api-75bd89fdfc-htl5n\" (UID: \"8c746478-bdc8-45c9-b5da-39a1fe98f1d1\") " pod="opendatahub/model-serving-api-75bd89fdfc-htl5n" Oct 30 20:43:19.038322 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:43:19.038280 4848 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Oct 30 20:43:19.038541 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:43:19.038354 4848 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26-cert podName:ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26 nodeName:}" failed. No retries permitted until 2026-10-30 20:43:19.538334499 +0000 UTC m=+304.718231002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26-cert") pod "odh-model-controller-769875db56-btqgb" (UID: "ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26") : secret "odh-model-controller-webhook-cert" not found Oct 30 20:43:19.038541 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.038390 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtpc8\" (UniqueName: \"kubernetes.io/projected/30fe2f63-49a7-43f5-9530-9f1cf25d635c-kube-api-access-vtpc8\") pod \"kserve-controller-manager-776cc66d47-chpzx\" (UID: \"30fe2f63-49a7-43f5-9530-9f1cf25d635c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-chpzx" Oct 30 20:43:19.038541 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.038429 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26-metrics-certs\") pod \"odh-model-controller-769875db56-btqgb\" (UID: \"ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26\") " pod="opendatahub/odh-model-controller-769875db56-btqgb" Oct 30 20:43:19.038541 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.038465 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c746478-bdc8-45c9-b5da-39a1fe98f1d1-tls-certs\") pod \"model-serving-api-75bd89fdfc-htl5n\" (UID: \"8c746478-bdc8-45c9-b5da-39a1fe98f1d1\") " pod="opendatahub/model-serving-api-75bd89fdfc-htl5n" Oct 30 20:43:19.046160 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.046116 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/30fe2f63-49a7-43f5-9530-9f1cf25d635c-cert\") pod \"kserve-controller-manager-776cc66d47-chpzx\" (UID: \"30fe2f63-49a7-43f5-9530-9f1cf25d635c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-chpzx" Oct 30 20:43:19.046306 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.046280 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26-metrics-certs\") pod \"odh-model-controller-769875db56-btqgb\" (UID: \"ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26\") " pod="opendatahub/odh-model-controller-769875db56-btqgb" Oct 30 20:43:19.046453 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.046432 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c746478-bdc8-45c9-b5da-39a1fe98f1d1-tls-certs\") pod \"model-serving-api-75bd89fdfc-htl5n\" (UID: \"8c746478-bdc8-45c9-b5da-39a1fe98f1d1\") " pod="opendatahub/model-serving-api-75bd89fdfc-htl5n" Oct 30 20:43:19.053640 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.053609 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtpc8\" (UniqueName: \"kubernetes.io/projected/30fe2f63-49a7-43f5-9530-9f1cf25d635c-kube-api-access-vtpc8\") pod \"kserve-controller-manager-776cc66d47-chpzx\" (UID: \"30fe2f63-49a7-43f5-9530-9f1cf25d635c\") " pod="opendatahub/kserve-controller-manager-776cc66d47-chpzx" Oct 30 20:43:19.053750 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.053607 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtqft\" (UniqueName: \"kubernetes.io/projected/ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26-kube-api-access-rtqft\") pod \"odh-model-controller-769875db56-btqgb\" (UID: \"ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26\") " pod="opendatahub/odh-model-controller-769875db56-btqgb" Oct 30 20:43:19.053813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.053746 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xh67\" (UniqueName: \"kubernetes.io/projected/8c746478-bdc8-45c9-b5da-39a1fe98f1d1-kube-api-access-7xh67\") pod \"model-serving-api-75bd89fdfc-htl5n\" (UID: \"8c746478-bdc8-45c9-b5da-39a1fe98f1d1\") " pod="opendatahub/model-serving-api-75bd89fdfc-htl5n" Oct 30 20:43:19.159218 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.159128 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-chpzx" Oct 30 20:43:19.203638 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.203601 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr"] Oct 30 20:43:19.208894 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.208123 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.213447 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.212944 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-htl5n" Oct 30 20:43:19.227822 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.227334 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr"] Oct 30 20:43:19.240666 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.240407 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.240666 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.240445 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f478d7b-4805-4559-93cb-33e622b3a537-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.240666 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.240477 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk46c\" (UniqueName: \"kubernetes.io/projected/1f478d7b-4805-4559-93cb-33e622b3a537-kube-api-access-mk46c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.240666 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.240492 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.240666 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.240514 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f478d7b-4805-4559-93cb-33e622b3a537-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.240666 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.240532 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f478d7b-4805-4559-93cb-33e622b3a537-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.240666 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.240546 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.341239 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.341204 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk46c\" (UniqueName: \"kubernetes.io/projected/1f478d7b-4805-4559-93cb-33e622b3a537-kube-api-access-mk46c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.341239 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.341246 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.341498 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.341283 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f478d7b-4805-4559-93cb-33e622b3a537-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.341498 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.341308 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f478d7b-4805-4559-93cb-33e622b3a537-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.341498 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.341332 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.341498 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.341411 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.341498 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.341440 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f478d7b-4805-4559-93cb-33e622b3a537-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.342189 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.342162 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f478d7b-4805-4559-93cb-33e622b3a537-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.343656 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.343634 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.343758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.343707 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.344435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.344409 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f478d7b-4805-4559-93cb-33e622b3a537-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.344515 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.344451 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.351828 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.351802 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f478d7b-4805-4559-93cb-33e622b3a537-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.352013 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.351997 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk46c\" (UniqueName: \"kubernetes.io/projected/1f478d7b-4805-4559-93cb-33e622b3a537-kube-api-access-mk46c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qm4kr\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.432116 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.432057 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-chpzx"] Oct 30 20:43:19.434693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:43:19.434666 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30fe2f63_49a7_43f5_9530_9f1cf25d635c.slice/crio-03375ef5c5da35db1f1a8369004e018359d12f3f5592c1aebe5a93851b56a7ed WatchSource:0}: Error finding container 03375ef5c5da35db1f1a8369004e018359d12f3f5592c1aebe5a93851b56a7ed: Status 404 returned error can't find the container with id 03375ef5c5da35db1f1a8369004e018359d12f3f5592c1aebe5a93851b56a7ed Oct 30 20:43:19.514176 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.514151 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-htl5n"] Oct 30 20:43:19.516567 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:43:19.516545 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c746478_bdc8_45c9_b5da_39a1fe98f1d1.slice/crio-4505c5567b6db89f7c829f43f7468f7dcb267bd06eb1491d2271f9cb00cbb01f WatchSource:0}: Error finding container 4505c5567b6db89f7c829f43f7468f7dcb267bd06eb1491d2271f9cb00cbb01f: Status 404 returned error can't find the container with id 4505c5567b6db89f7c829f43f7468f7dcb267bd06eb1491d2271f9cb00cbb01f Oct 30 20:43:19.539213 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.539187 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:19.543245 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.543136 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26-cert\") pod \"odh-model-controller-769875db56-btqgb\" (UID: \"ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26\") " pod="opendatahub/odh-model-controller-769875db56-btqgb" Oct 30 20:43:19.545615 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.545587 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26-cert\") pod \"odh-model-controller-769875db56-btqgb\" (UID: \"ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26\") " pod="opendatahub/odh-model-controller-769875db56-btqgb" Oct 30 20:43:19.827893 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.827817 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr"] Oct 30 20:43:19.830817 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.830793 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-btqgb" Oct 30 20:43:19.831221 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:43:19.831197 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f478d7b_4805_4559_93cb_33e622b3a537.slice/crio-fe6a988f5773e981e40a04aeac95575d1fae9ab31bef2ec2bc6beea80b9ff554 WatchSource:0}: Error finding container fe6a988f5773e981e40a04aeac95575d1fae9ab31bef2ec2bc6beea80b9ff554: Status 404 returned error can't find the container with id fe6a988f5773e981e40a04aeac95575d1fae9ab31bef2ec2bc6beea80b9ff554 Oct 30 20:43:19.833252 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.833201 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:43:19.833337 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:19.833281 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:43:20.018863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:20.018815 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-chpzx" event={"ID":"30fe2f63-49a7-43f5-9530-9f1cf25d635c","Type":"ContainerStarted","Data":"03375ef5c5da35db1f1a8369004e018359d12f3f5592c1aebe5a93851b56a7ed"} Oct 30 20:43:20.020349 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:20.020323 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-htl5n" event={"ID":"8c746478-bdc8-45c9-b5da-39a1fe98f1d1","Type":"ContainerStarted","Data":"4505c5567b6db89f7c829f43f7468f7dcb267bd06eb1491d2271f9cb00cbb01f"} Oct 30 20:43:20.024211 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:20.022965 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" event={"ID":"1f478d7b-4805-4559-93cb-33e622b3a537","Type":"ContainerStarted","Data":"aa1c06a37cef2b28eafb1a2742ded5535b1745645655f180451bb671526a8a7c"} Oct 30 20:43:20.024211 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:20.023011 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" event={"ID":"1f478d7b-4805-4559-93cb-33e622b3a537","Type":"ContainerStarted","Data":"fe6a988f5773e981e40a04aeac95575d1fae9ab31bef2ec2bc6beea80b9ff554"} Oct 30 20:43:20.024211 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:20.024162 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:20.025932 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:20.025895 4848 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-qm4kr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:43:20.026033 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:20.025964 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" podUID="1f478d7b-4805-4559-93cb-33e622b3a537" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:43:20.109954 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:20.109845 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" podStartSLOduration=1.109829239 podStartE2EDuration="1.109829239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:20.048861539 +0000 UTC m=+305.228758047" watchObservedRunningTime="2026-10-30 20:43:20.109829239 +0000 UTC m=+305.289725737" Oct 30 20:43:20.111023 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:20.110942 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-btqgb"] Oct 30 20:43:20.116893 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:43:20.116859 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0ac0d4_5487_4f29_8a5a_ffe74ef87d26.slice/crio-469a6142fb799c4a9a2d032b57ba9dfa886c012f89fd7c7438bb05333c6365b2 WatchSource:0}: Error finding container 469a6142fb799c4a9a2d032b57ba9dfa886c012f89fd7c7438bb05333c6365b2: Status 404 returned error can't find the container with id 469a6142fb799c4a9a2d032b57ba9dfa886c012f89fd7c7438bb05333c6365b2 Oct 30 20:43:20.853813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:20.852961 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jpzw8"] Oct 30 20:43:20.853813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:20.853215 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-jpzw8" podUID="7d2f37d2-e737-4f6b-be8a-c36c5bead2b6" containerName="limitador" containerID="cri-o://96a3044e8c159d511c41fd34e9aa33d0d42bdb33a23db06fc85c73d6f94bb664" gracePeriod=30 Oct 30 20:43:20.855003 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:20.854975 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-jpzw8" Oct 30 20:43:21.032421 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.032105 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-btqgb" event={"ID":"ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26","Type":"ContainerStarted","Data":"469a6142fb799c4a9a2d032b57ba9dfa886c012f89fd7c7438bb05333c6365b2"} Oct 30 20:43:21.033173 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.032984 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 20:43:21.122373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.122300 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc"] Oct 30 20:43:21.122526 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.122489 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" podUID="5ed2dc2e-2e21-4402-ac8f-0101c97cfc89" containerName="discovery" containerID="cri-o://ab72f32f5265a199a346afafd6fd9f95461a88786848b515589f4f672b6ae2e1" gracePeriod=30 Oct 30 20:43:21.432948 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.432654 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:43:21.441179 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.441154 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jpzw8" Oct 30 20:43:21.465525 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.463341 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-kube-api-access-gctlf\" (UniqueName: \"kubernetes.io/projected/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-kube-api-access-gctlf\") pod \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " Oct 30 20:43:21.465525 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.463394 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2f37d2-e737-4f6b-be8a-c36c5bead2b6-config-file\" (UniqueName: \"kubernetes.io/configmap/7d2f37d2-e737-4f6b-be8a-c36c5bead2b6-config-file\") pod \"7d2f37d2-e737-4f6b-be8a-c36c5bead2b6\" (UID: \"7d2f37d2-e737-4f6b-be8a-c36c5bead2b6\") " Oct 30 20:43:21.465525 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.463449 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-token\" (UniqueName: \"kubernetes.io/projected/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-token\") pod \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " Oct 30 20:43:21.465525 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.463488 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2f37d2-e737-4f6b-be8a-c36c5bead2b6-kube-api-access-ctb6d\" (UniqueName: \"kubernetes.io/projected/7d2f37d2-e737-4f6b-be8a-c36c5bead2b6-kube-api-access-ctb6d\") pod \"7d2f37d2-e737-4f6b-be8a-c36c5bead2b6\" (UID: \"7d2f37d2-e737-4f6b-be8a-c36c5bead2b6\") " Oct 30 20:43:21.465525 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.463564 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-csr-dns-cert\") pod \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " Oct 30 20:43:21.465525 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.463594 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-cacerts\" (UniqueName: \"kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-cacerts\") pod \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " Oct 30 20:43:21.465525 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.463631 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-local-certs\") pod \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " Oct 30 20:43:21.465525 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.463659 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-csr-ca-configmap\") pod \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " Oct 30 20:43:21.465525 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.463686 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-kubeconfig\") pod \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\" (UID: \"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89\") " Oct 30 20:43:21.465525 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.464832 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2f37d2-e737-4f6b-be8a-c36c5bead2b6-config-file" pod "7d2f37d2-e737-4f6b-be8a-c36c5bead2b6" (UID: "7d2f37d2-e737-4f6b-be8a-c36c5bead2b6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:43:21.466236 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.466105 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-csr-ca-configmap" pod "5ed2dc2e-2e21-4402-ac8f-0101c97cfc89" (UID: "5ed2dc2e-2e21-4402-ac8f-0101c97cfc89"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:43:21.469664 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.469636 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-kube-api-access-gctlf" pod "5ed2dc2e-2e21-4402-ac8f-0101c97cfc89" (UID: "5ed2dc2e-2e21-4402-ac8f-0101c97cfc89"). InnerVolumeSpecName "kube-api-access-gctlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:43:21.470228 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.470195 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2f37d2-e737-4f6b-be8a-c36c5bead2b6-kube-api-access-ctb6d" pod "7d2f37d2-e737-4f6b-be8a-c36c5bead2b6" (UID: "7d2f37d2-e737-4f6b-be8a-c36c5bead2b6"). InnerVolumeSpecName "kube-api-access-ctb6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:43:21.470412 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.470354 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-kubeconfig" pod "5ed2dc2e-2e21-4402-ac8f-0101c97cfc89" (UID: "5ed2dc2e-2e21-4402-ac8f-0101c97cfc89"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:43:21.470759 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.470733 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-token" pod "5ed2dc2e-2e21-4402-ac8f-0101c97cfc89" (UID: "5ed2dc2e-2e21-4402-ac8f-0101c97cfc89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:43:21.470922 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.470900 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-local-certs" pod "5ed2dc2e-2e21-4402-ac8f-0101c97cfc89" (UID: "5ed2dc2e-2e21-4402-ac8f-0101c97cfc89"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:43:21.472141 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.472114 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-csr-dns-cert" pod "5ed2dc2e-2e21-4402-ac8f-0101c97cfc89" (UID: "5ed2dc2e-2e21-4402-ac8f-0101c97cfc89"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:43:21.474844 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.474806 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-cacerts" pod "5ed2dc2e-2e21-4402-ac8f-0101c97cfc89" (UID: "5ed2dc2e-2e21-4402-ac8f-0101c97cfc89"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:43:21.565153 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.565114 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:43:21.565319 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.565159 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctb6d\" (UniqueName: \"kubernetes.io/projected/7d2f37d2-e737-4f6b-be8a-c36c5bead2b6-kube-api-access-ctb6d\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:43:21.565319 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.565177 4848 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-csr-dns-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:43:21.565319 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.565191 4848 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-cacerts\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:43:21.565319 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.565205 4848 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-local-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:43:21.565319 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.565219 4848 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-csr-ca-configmap\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:43:21.565319 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.565233 4848 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-istio-kubeconfig\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:43:21.565319 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.565246 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gctlf\" (UniqueName: \"kubernetes.io/projected/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89-kube-api-access-gctlf\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:43:21.565319 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:21.565260 4848 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d2f37d2-e737-4f6b-be8a-c36c5bead2b6-config-file\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:43:22.038412 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.038331 4848 generic.go:356] "Generic (PLEG): container finished" podID="5ed2dc2e-2e21-4402-ac8f-0101c97cfc89" containerID="ab72f32f5265a199a346afafd6fd9f95461a88786848b515589f4f672b6ae2e1" exitCode=0 Oct 30 20:43:22.038865 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.038513 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" Oct 30 20:43:22.038940 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.038911 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" event={"ID":"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89","Type":"ContainerDied","Data":"ab72f32f5265a199a346afafd6fd9f95461a88786848b515589f4f672b6ae2e1"} Oct 30 20:43:22.039010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.038969 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc" event={"ID":"5ed2dc2e-2e21-4402-ac8f-0101c97cfc89","Type":"ContainerDied","Data":"9abcc7e7dd29743fd39af09be986195789bb96383680983b44d505ff4dad5ae3"} Oct 30 20:43:22.039010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.038992 4848 scope.go:122] "RemoveContainer" containerID="ab72f32f5265a199a346afafd6fd9f95461a88786848b515589f4f672b6ae2e1" Oct 30 20:43:22.041581 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.041184 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-htl5n" event={"ID":"8c746478-bdc8-45c9-b5da-39a1fe98f1d1","Type":"ContainerStarted","Data":"db624843e0af641860c78a7a983768fd985c0faf0e7f874ab1d312f0f58eca3b"} Oct 30 20:43:22.041581 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.041346 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-htl5n" Oct 30 20:43:22.042612 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.042590 4848 generic.go:356] "Generic (PLEG): container finished" podID="7d2f37d2-e737-4f6b-be8a-c36c5bead2b6" containerID="96a3044e8c159d511c41fd34e9aa33d0d42bdb33a23db06fc85c73d6f94bb664" exitCode=0 Oct 30 20:43:22.042717 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.042628 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jpzw8" Oct 30 20:43:22.042717 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.042659 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jpzw8" event={"ID":"7d2f37d2-e737-4f6b-be8a-c36c5bead2b6","Type":"ContainerDied","Data":"96a3044e8c159d511c41fd34e9aa33d0d42bdb33a23db06fc85c73d6f94bb664"} Oct 30 20:43:22.042717 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.042692 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jpzw8" event={"ID":"7d2f37d2-e737-4f6b-be8a-c36c5bead2b6","Type":"ContainerDied","Data":"d91efc7248e440bccd5a8140448cb4650da4d9bf6c44e5823482dda28cb2a4f6"} Oct 30 20:43:22.051116 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.051077 4848 scope.go:122] "RemoveContainer" containerID="ab72f32f5265a199a346afafd6fd9f95461a88786848b515589f4f672b6ae2e1" Oct 30 20:43:22.051407 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:43:22.051383 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab72f32f5265a199a346afafd6fd9f95461a88786848b515589f4f672b6ae2e1\": container with ID starting with ab72f32f5265a199a346afafd6fd9f95461a88786848b515589f4f672b6ae2e1 not found: ID does not exist" containerID="ab72f32f5265a199a346afafd6fd9f95461a88786848b515589f4f672b6ae2e1" Oct 30 20:43:22.051490 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.051417 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab72f32f5265a199a346afafd6fd9f95461a88786848b515589f4f672b6ae2e1"} err="failed to get container status \"ab72f32f5265a199a346afafd6fd9f95461a88786848b515589f4f672b6ae2e1\": rpc error: code = NotFound desc = could not find container \"ab72f32f5265a199a346afafd6fd9f95461a88786848b515589f4f672b6ae2e1\": container with ID starting with ab72f32f5265a199a346afafd6fd9f95461a88786848b515589f4f672b6ae2e1 not found: ID does not exist" Oct 30 20:43:22.051490 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.051442 4848 scope.go:122] "RemoveContainer" containerID="96a3044e8c159d511c41fd34e9aa33d0d42bdb33a23db06fc85c73d6f94bb664" Oct 30 20:43:22.060367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.060344 4848 scope.go:122] "RemoveContainer" containerID="96a3044e8c159d511c41fd34e9aa33d0d42bdb33a23db06fc85c73d6f94bb664" Oct 30 20:43:22.060680 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:43:22.060653 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96a3044e8c159d511c41fd34e9aa33d0d42bdb33a23db06fc85c73d6f94bb664\": container with ID starting with 96a3044e8c159d511c41fd34e9aa33d0d42bdb33a23db06fc85c73d6f94bb664 not found: ID does not exist" containerID="96a3044e8c159d511c41fd34e9aa33d0d42bdb33a23db06fc85c73d6f94bb664" Oct 30 20:43:22.060839 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.060802 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96a3044e8c159d511c41fd34e9aa33d0d42bdb33a23db06fc85c73d6f94bb664"} err="failed to get container status \"96a3044e8c159d511c41fd34e9aa33d0d42bdb33a23db06fc85c73d6f94bb664\": rpc error: code = NotFound desc = could not find container \"96a3044e8c159d511c41fd34e9aa33d0d42bdb33a23db06fc85c73d6f94bb664\": container with ID starting with 96a3044e8c159d511c41fd34e9aa33d0d42bdb33a23db06fc85c73d6f94bb664 not found: ID does not exist" Oct 30 20:43:22.067559 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.066579 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-htl5n" podStartSLOduration=1.853469845 podStartE2EDuration="4.066567347s" totalImagesPullingTime="2.213097502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:22.066144315 +0000 UTC m=+307.246040836" watchObservedRunningTime="2026-10-30 20:43:22.066567347 +0000 UTC m=+307.246463862" Oct 30 20:43:22.083145 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.083114 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc"] Oct 30 20:43:22.108170 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.108138 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jqmqc"] Oct 30 20:43:22.127533 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.127507 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jpzw8"] Oct 30 20:43:22.132707 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.132684 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jpzw8"] Oct 30 20:43:22.514363 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:43:22.514293 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:43:22.930654 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.930621 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ed2dc2e-2e21-4402-ac8f-0101c97cfc89" path="/var/lib/kubelet/pods/5ed2dc2e-2e21-4402-ac8f-0101c97cfc89/volumes" Oct 30 20:43:22.931126 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:22.931109 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d2f37d2-e737-4f6b-be8a-c36c5bead2b6" path="/var/lib/kubelet/pods/7d2f37d2-e737-4f6b-be8a-c36c5bead2b6/volumes" Oct 30 20:43:23.052715 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:23.052677 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-chpzx" event={"ID":"30fe2f63-49a7-43f5-9530-9f1cf25d635c","Type":"ContainerStarted","Data":"94e4e2c46c03c00851295b10873eeec512ad5fe0b8077dfef2ddd3bbce3156fb"} Oct 30 20:43:23.074635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:23.074579 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-chpzx" podStartSLOduration=2.380793096 podStartE2EDuration="5.074564887s" totalImagesPullingTime="2.693771791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:23.073523693 +0000 UTC m=+308.253420201" watchObservedRunningTime="2026-10-30 20:43:23.074564887 +0000 UTC m=+308.254461393" Oct 30 20:43:24.058492 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:24.058452 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-btqgb" event={"ID":"ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26","Type":"ContainerStarted","Data":"50658bde92add409eef1aebb29f68bc5c5889b9ea53f0856b1503d6a173b55b1"} Oct 30 20:43:24.058903 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:24.058628 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-btqgb" Oct 30 20:43:24.058903 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:24.058805 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-chpzx" Oct 30 20:43:24.079512 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:24.079465 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-btqgb" podStartSLOduration=3.049364222 podStartE2EDuration="6.079451959s" totalImagesPullingTime="3.030087737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:24.078656005 +0000 UTC m=+309.258552512" watchObservedRunningTime="2026-10-30 20:43:24.079451959 +0000 UTC m=+309.259348513" Oct 30 20:43:26.896208 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:26.896172 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-b28mc"] Oct 30 20:43:26.896591 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:26.896492 4848 cpu_manager.go:414] "containerMap: removing container" podUID="5ed2dc2e-2e21-4402-ac8f-0101c97cfc89" containerName="discovery" Oct 30 20:43:26.896591 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:26.896503 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed2dc2e-2e21-4402-ac8f-0101c97cfc89" containerName="discovery" Oct 30 20:43:26.896591 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:26.896514 4848 cpu_manager.go:414] "containerMap: removing container" podUID="7d2f37d2-e737-4f6b-be8a-c36c5bead2b6" containerName="limitador" Oct 30 20:43:26.896591 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:26.896519 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2f37d2-e737-4f6b-be8a-c36c5bead2b6" containerName="limitador" Oct 30 20:43:26.896591 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:26.896581 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ed2dc2e-2e21-4402-ac8f-0101c97cfc89" containerName="discovery" Oct 30 20:43:26.896591 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:26.896589 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2f37d2-e737-4f6b-be8a-c36c5bead2b6" containerName="limitador" Oct 30 20:43:26.899635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:26.899619 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b28mc" Oct 30 20:43:26.919618 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:26.919584 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 20:43:26.919900 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:26.919881 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-rzhlj\"" Oct 30 20:43:26.934473 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:26.934432 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:43:26.934606 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:26.934545 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:43:26.952470 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:26.952446 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-b28mc"] Oct 30 20:43:27.008156 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:27.008126 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-476vs\" (UniqueName: \"kubernetes.io/projected/fe4fdd20-b075-45d5-ae3d-3959bfa3bb51-kube-api-access-476vs\") pod \"postgres-7cdbf6dc44-b28mc\" (UID: \"fe4fdd20-b075-45d5-ae3d-3959bfa3bb51\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b28mc" Oct 30 20:43:27.008285 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:27.008176 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fe4fdd20-b075-45d5-ae3d-3959bfa3bb51-data\") pod \"postgres-7cdbf6dc44-b28mc\" (UID: \"fe4fdd20-b075-45d5-ae3d-3959bfa3bb51\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b28mc" Oct 30 20:43:27.109585 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:27.109552 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-476vs\" (UniqueName: \"kubernetes.io/projected/fe4fdd20-b075-45d5-ae3d-3959bfa3bb51-kube-api-access-476vs\") pod \"postgres-7cdbf6dc44-b28mc\" (UID: \"fe4fdd20-b075-45d5-ae3d-3959bfa3bb51\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b28mc" Oct 30 20:43:27.109783 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:27.109597 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fe4fdd20-b075-45d5-ae3d-3959bfa3bb51-data\") pod \"postgres-7cdbf6dc44-b28mc\" (UID: \"fe4fdd20-b075-45d5-ae3d-3959bfa3bb51\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b28mc" Oct 30 20:43:27.109966 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:27.109948 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fe4fdd20-b075-45d5-ae3d-3959bfa3bb51-data\") pod \"postgres-7cdbf6dc44-b28mc\" (UID: \"fe4fdd20-b075-45d5-ae3d-3959bfa3bb51\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b28mc" Oct 30 20:43:27.121300 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:27.121274 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-476vs\" (UniqueName: \"kubernetes.io/projected/fe4fdd20-b075-45d5-ae3d-3959bfa3bb51-kube-api-access-476vs\") pod \"postgres-7cdbf6dc44-b28mc\" (UID: \"fe4fdd20-b075-45d5-ae3d-3959bfa3bb51\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b28mc" Oct 30 20:43:27.211608 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:27.211572 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b28mc" Oct 30 20:43:27.494584 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:27.494509 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-b28mc"] Oct 30 20:43:27.497688 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:43:27.497660 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4fdd20_b075_45d5_ae3d_3959bfa3bb51.slice/crio-09ae0a83838ba08bc4297a425ee29fd9071a33a4548858442b591065c314f9b5 WatchSource:0}: Error finding container 09ae0a83838ba08bc4297a425ee29fd9071a33a4548858442b591065c314f9b5: Status 404 returned error can't find the container with id 09ae0a83838ba08bc4297a425ee29fd9071a33a4548858442b591065c314f9b5 Oct 30 20:43:28.074253 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:28.074217 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b28mc" event={"ID":"fe4fdd20-b075-45d5-ae3d-3959bfa3bb51","Type":"ContainerStarted","Data":"09ae0a83838ba08bc4297a425ee29fd9071a33a4548858442b591065c314f9b5"} Oct 30 20:43:33.056718 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:33.056645 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-htl5n" Oct 30 20:43:33.098221 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:33.098189 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b28mc" event={"ID":"fe4fdd20-b075-45d5-ae3d-3959bfa3bb51","Type":"ContainerStarted","Data":"5cb233a3442db0c59d986fb810745c327c19c46ed86465d10281999760f76ef7"} Oct 30 20:43:33.098392 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:33.098312 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b28mc" Oct 30 20:43:33.116573 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:33.116523 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b28mc" podStartSLOduration=1.849353778 podStartE2EDuration="7.116510564s" totalImagesPullingTime="5.267156786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:33.115375546 +0000 UTC m=+318.295272053" watchObservedRunningTime="2026-10-30 20:43:33.116510564 +0000 UTC m=+318.296407070" Oct 30 20:43:35.064884 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:35.064854 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-btqgb" Oct 30 20:43:39.131508 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:39.131476 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-b28mc" Oct 30 20:43:46.028415 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.028380 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67dcc64f86-rdbqv"] Oct 30 20:43:46.130562 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.130522 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67dcc64f86-rdbqv"] Oct 30 20:43:46.130562 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.130559 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67dcc64f86-rdbqv" Oct 30 20:43:46.135517 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.135493 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sb4qs\"" Oct 30 20:43:46.135517 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.135493 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 20:43:46.135701 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.135537 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 20:43:46.135701 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.135590 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 20:43:46.276142 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.276102 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fc2v\" (UniqueName: \"kubernetes.io/projected/7c9ef95b-439e-4657-ac41-b67752e7174c-kube-api-access-9fc2v\") pod \"maas-controller-67dcc64f86-rdbqv\" (UID: \"7c9ef95b-439e-4657-ac41-b67752e7174c\") " pod="opendatahub/maas-controller-67dcc64f86-rdbqv" Oct 30 20:43:46.276308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.276160 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c9ef95b-439e-4657-ac41-b67752e7174c-webhook-cert\") pod \"maas-controller-67dcc64f86-rdbqv\" (UID: \"7c9ef95b-439e-4657-ac41-b67752e7174c\") " pod="opendatahub/maas-controller-67dcc64f86-rdbqv" Oct 30 20:43:46.276308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.276186 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c9ef95b-439e-4657-ac41-b67752e7174c-metrics-certs\") pod \"maas-controller-67dcc64f86-rdbqv\" (UID: \"7c9ef95b-439e-4657-ac41-b67752e7174c\") " pod="opendatahub/maas-controller-67dcc64f86-rdbqv" Oct 30 20:43:46.376823 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.376722 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c9ef95b-439e-4657-ac41-b67752e7174c-webhook-cert\") pod \"maas-controller-67dcc64f86-rdbqv\" (UID: \"7c9ef95b-439e-4657-ac41-b67752e7174c\") " pod="opendatahub/maas-controller-67dcc64f86-rdbqv" Oct 30 20:43:46.376823 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.376782 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c9ef95b-439e-4657-ac41-b67752e7174c-metrics-certs\") pod \"maas-controller-67dcc64f86-rdbqv\" (UID: \"7c9ef95b-439e-4657-ac41-b67752e7174c\") " pod="opendatahub/maas-controller-67dcc64f86-rdbqv" Oct 30 20:43:46.377005 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.376836 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fc2v\" (UniqueName: \"kubernetes.io/projected/7c9ef95b-439e-4657-ac41-b67752e7174c-kube-api-access-9fc2v\") pod \"maas-controller-67dcc64f86-rdbqv\" (UID: \"7c9ef95b-439e-4657-ac41-b67752e7174c\") " pod="opendatahub/maas-controller-67dcc64f86-rdbqv" Oct 30 20:43:46.379263 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.379229 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c9ef95b-439e-4657-ac41-b67752e7174c-webhook-cert\") pod \"maas-controller-67dcc64f86-rdbqv\" (UID: \"7c9ef95b-439e-4657-ac41-b67752e7174c\") " pod="opendatahub/maas-controller-67dcc64f86-rdbqv" Oct 30 20:43:46.379376 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.379278 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c9ef95b-439e-4657-ac41-b67752e7174c-metrics-certs\") pod \"maas-controller-67dcc64f86-rdbqv\" (UID: \"7c9ef95b-439e-4657-ac41-b67752e7174c\") " pod="opendatahub/maas-controller-67dcc64f86-rdbqv" Oct 30 20:43:46.386413 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.386389 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fc2v\" (UniqueName: \"kubernetes.io/projected/7c9ef95b-439e-4657-ac41-b67752e7174c-kube-api-access-9fc2v\") pod \"maas-controller-67dcc64f86-rdbqv\" (UID: \"7c9ef95b-439e-4657-ac41-b67752e7174c\") " pod="opendatahub/maas-controller-67dcc64f86-rdbqv" Oct 30 20:43:46.453337 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.453297 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67dcc64f86-rdbqv" Oct 30 20:43:46.504443 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.504408 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f5d6dd878-twxgq"] Oct 30 20:43:46.511128 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.511103 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f5d6dd878-twxgq" Oct 30 20:43:46.520009 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.519560 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f5d6dd878-twxgq"] Oct 30 20:43:46.679187 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.679152 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44vgx\" (UniqueName: \"kubernetes.io/projected/e1ef8eca-79af-4e64-8cba-4b9caaa7e574-kube-api-access-44vgx\") pod \"maas-controller-f5d6dd878-twxgq\" (UID: \"e1ef8eca-79af-4e64-8cba-4b9caaa7e574\") " pod="opendatahub/maas-controller-f5d6dd878-twxgq" Oct 30 20:43:46.679386 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.679191 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1ef8eca-79af-4e64-8cba-4b9caaa7e574-metrics-certs\") pod \"maas-controller-f5d6dd878-twxgq\" (UID: \"e1ef8eca-79af-4e64-8cba-4b9caaa7e574\") " pod="opendatahub/maas-controller-f5d6dd878-twxgq" Oct 30 20:43:46.679386 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.679287 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1ef8eca-79af-4e64-8cba-4b9caaa7e574-webhook-cert\") pod \"maas-controller-f5d6dd878-twxgq\" (UID: \"e1ef8eca-79af-4e64-8cba-4b9caaa7e574\") " pod="opendatahub/maas-controller-f5d6dd878-twxgq" Oct 30 20:43:46.765635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.765596 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67dcc64f86-rdbqv"] Oct 30 20:43:46.768571 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:43:46.768545 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c9ef95b_439e_4657_ac41_b67752e7174c.slice/crio-feb097324ee168b70c447dac665076254e921dff3acead1acf881209f79e88ea WatchSource:0}: Error finding container feb097324ee168b70c447dac665076254e921dff3acead1acf881209f79e88ea: Status 404 returned error can't find the container with id feb097324ee168b70c447dac665076254e921dff3acead1acf881209f79e88ea Oct 30 20:43:46.769898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.769879 4848 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:43:46.780052 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.780029 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44vgx\" (UniqueName: \"kubernetes.io/projected/e1ef8eca-79af-4e64-8cba-4b9caaa7e574-kube-api-access-44vgx\") pod \"maas-controller-f5d6dd878-twxgq\" (UID: \"e1ef8eca-79af-4e64-8cba-4b9caaa7e574\") " pod="opendatahub/maas-controller-f5d6dd878-twxgq" Oct 30 20:43:46.780138 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.780059 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1ef8eca-79af-4e64-8cba-4b9caaa7e574-metrics-certs\") pod \"maas-controller-f5d6dd878-twxgq\" (UID: \"e1ef8eca-79af-4e64-8cba-4b9caaa7e574\") " pod="opendatahub/maas-controller-f5d6dd878-twxgq" Oct 30 20:43:46.780138 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.780080 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1ef8eca-79af-4e64-8cba-4b9caaa7e574-webhook-cert\") pod \"maas-controller-f5d6dd878-twxgq\" (UID: \"e1ef8eca-79af-4e64-8cba-4b9caaa7e574\") " pod="opendatahub/maas-controller-f5d6dd878-twxgq" Oct 30 20:43:46.782414 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.782394 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1ef8eca-79af-4e64-8cba-4b9caaa7e574-webhook-cert\") pod \"maas-controller-f5d6dd878-twxgq\" (UID: \"e1ef8eca-79af-4e64-8cba-4b9caaa7e574\") " pod="opendatahub/maas-controller-f5d6dd878-twxgq" Oct 30 20:43:46.782475 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.782436 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1ef8eca-79af-4e64-8cba-4b9caaa7e574-metrics-certs\") pod \"maas-controller-f5d6dd878-twxgq\" (UID: \"e1ef8eca-79af-4e64-8cba-4b9caaa7e574\") " pod="opendatahub/maas-controller-f5d6dd878-twxgq" Oct 30 20:43:46.788629 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.788610 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44vgx\" (UniqueName: \"kubernetes.io/projected/e1ef8eca-79af-4e64-8cba-4b9caaa7e574-kube-api-access-44vgx\") pod \"maas-controller-f5d6dd878-twxgq\" (UID: \"e1ef8eca-79af-4e64-8cba-4b9caaa7e574\") " pod="opendatahub/maas-controller-f5d6dd878-twxgq" Oct 30 20:43:46.842640 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:46.842605 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f5d6dd878-twxgq" Oct 30 20:43:47.160442 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:47.160392 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67dcc64f86-rdbqv" event={"ID":"7c9ef95b-439e-4657-ac41-b67752e7174c","Type":"ContainerStarted","Data":"feb097324ee168b70c447dac665076254e921dff3acead1acf881209f79e88ea"} Oct 30 20:43:47.161823 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:47.161800 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f5d6dd878-twxgq"] Oct 30 20:43:47.164306 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:43:47.164285 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ef8eca_79af_4e64_8cba_4b9caaa7e574.slice/crio-5bf6cad96d0230f4e73f8b373b745feb2ac578c91d1d41ec6c8ae540f21da7d1 WatchSource:0}: Error finding container 5bf6cad96d0230f4e73f8b373b745feb2ac578c91d1d41ec6c8ae540f21da7d1: Status 404 returned error can't find the container with id 5bf6cad96d0230f4e73f8b373b745feb2ac578c91d1d41ec6c8ae540f21da7d1 Oct 30 20:43:48.166083 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:48.166045 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f5d6dd878-twxgq" event={"ID":"e1ef8eca-79af-4e64-8cba-4b9caaa7e574","Type":"ContainerStarted","Data":"5bf6cad96d0230f4e73f8b373b745feb2ac578c91d1d41ec6c8ae540f21da7d1"} Oct 30 20:43:52.185831 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:52.185796 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f5d6dd878-twxgq" event={"ID":"e1ef8eca-79af-4e64-8cba-4b9caaa7e574","Type":"ContainerStarted","Data":"982e2ff7a3afb5086451d82e49f5287228ae04125b5b2b7640116ee4aecf525d"} Oct 30 20:43:52.186309 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:52.185909 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f5d6dd878-twxgq" Oct 30 20:43:52.187253 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:52.187233 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67dcc64f86-rdbqv" event={"ID":"7c9ef95b-439e-4657-ac41-b67752e7174c","Type":"ContainerStarted","Data":"d1576d5086fa15b1ec827d6583ac75854dd36019ee78fa96a75f1ca55649b817"} Oct 30 20:43:52.187372 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:52.187356 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67dcc64f86-rdbqv" Oct 30 20:43:52.205479 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:52.205414 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-f5d6dd878-twxgq" podStartSLOduration=1.285936809 podStartE2EDuration="6.205394771s" totalImagesPullingTime="4.919457962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:52.203149273 +0000 UTC m=+337.383045782" watchObservedRunningTime="2026-10-30 20:43:52.205394771 +0000 UTC m=+337.385291278" Oct 30 20:43:52.222281 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:52.222226 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-67dcc64f86-rdbqv" podStartSLOduration=0.899891723 podStartE2EDuration="6.222209722s" totalImagesPullingTime="5.322317999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:43:52.220643567 +0000 UTC m=+337.400540078" watchObservedRunningTime="2026-10-30 20:43:52.222209722 +0000 UTC m=+337.402106312" Oct 30 20:43:52.510394 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:43:52.510354 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:43:53.849909 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:53.849877 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l"] Oct 30 20:43:53.852621 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:53.852597 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l" Oct 30 20:43:53.857715 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:53.857684 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-jblq9\"" Oct 30 20:43:53.857902 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:53.857779 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 20:43:53.857902 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:53.857883 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 20:43:53.858010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:53.857926 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 20:43:53.864521 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:53.864490 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l"] Oct 30 20:43:53.944611 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:53.944575 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7dce4323-1225-4bf8-b93e-a2a482b657a5-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rf52l\" (UID: \"7dce4323-1225-4bf8-b93e-a2a482b657a5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l" Oct 30 20:43:53.944611 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:53.944616 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/7dce4323-1225-4bf8-b93e-a2a482b657a5-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rf52l\" (UID: \"7dce4323-1225-4bf8-b93e-a2a482b657a5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l" Oct 30 20:43:53.944859 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:53.944738 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7dce4323-1225-4bf8-b93e-a2a482b657a5-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rf52l\" (UID: \"7dce4323-1225-4bf8-b93e-a2a482b657a5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l" Oct 30 20:43:53.944859 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:53.944786 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpvp5\" (UniqueName: \"kubernetes.io/projected/7dce4323-1225-4bf8-b93e-a2a482b657a5-kube-api-access-gpvp5\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rf52l\" (UID: \"7dce4323-1225-4bf8-b93e-a2a482b657a5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l" Oct 30 20:43:54.046176 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:54.046143 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7dce4323-1225-4bf8-b93e-a2a482b657a5-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rf52l\" (UID: \"7dce4323-1225-4bf8-b93e-a2a482b657a5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l" Oct 30 20:43:54.046176 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:54.046182 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/7dce4323-1225-4bf8-b93e-a2a482b657a5-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rf52l\" (UID: \"7dce4323-1225-4bf8-b93e-a2a482b657a5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l" Oct 30 20:43:54.046397 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:54.046259 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7dce4323-1225-4bf8-b93e-a2a482b657a5-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rf52l\" (UID: \"7dce4323-1225-4bf8-b93e-a2a482b657a5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l" Oct 30 20:43:54.046397 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:54.046285 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpvp5\" (UniqueName: \"kubernetes.io/projected/7dce4323-1225-4bf8-b93e-a2a482b657a5-kube-api-access-gpvp5\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rf52l\" (UID: \"7dce4323-1225-4bf8-b93e-a2a482b657a5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l" Oct 30 20:43:54.047127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:54.047101 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7dce4323-1225-4bf8-b93e-a2a482b657a5-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rf52l\" (UID: \"7dce4323-1225-4bf8-b93e-a2a482b657a5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l" Oct 30 20:43:54.047314 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:54.047296 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/7dce4323-1225-4bf8-b93e-a2a482b657a5-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rf52l\" (UID: \"7dce4323-1225-4bf8-b93e-a2a482b657a5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l" Oct 30 20:43:54.048657 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:54.048627 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7dce4323-1225-4bf8-b93e-a2a482b657a5-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rf52l\" (UID: \"7dce4323-1225-4bf8-b93e-a2a482b657a5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l" Oct 30 20:43:54.055486 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:54.055462 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpvp5\" (UniqueName: \"kubernetes.io/projected/7dce4323-1225-4bf8-b93e-a2a482b657a5-kube-api-access-gpvp5\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-rf52l\" (UID: \"7dce4323-1225-4bf8-b93e-a2a482b657a5\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l" Oct 30 20:43:54.166832 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:54.166729 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l" Oct 30 20:43:54.440340 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:54.440308 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l"] Oct 30 20:43:54.444098 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:43:54.444068 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dce4323_1225_4bf8_b93e_a2a482b657a5.slice/crio-81ede3db6da270e107a0b5e77f0e032c4d9dadd403722924454cdfd11be3003c WatchSource:0}: Error finding container 81ede3db6da270e107a0b5e77f0e032c4d9dadd403722924454cdfd11be3003c: Status 404 returned error can't find the container with id 81ede3db6da270e107a0b5e77f0e032c4d9dadd403722924454cdfd11be3003c Oct 30 20:43:55.067450 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:55.067424 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-chpzx" Oct 30 20:43:55.202841 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:43:55.202810 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l" event={"ID":"7dce4323-1225-4bf8-b93e-a2a482b657a5","Type":"ContainerStarted","Data":"81ede3db6da270e107a0b5e77f0e032c4d9dadd403722924454cdfd11be3003c"} Oct 30 20:44:03.194061 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:03.194023 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67dcc64f86-rdbqv" Oct 30 20:44:03.194468 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:03.194104 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f5d6dd878-twxgq" Oct 30 20:44:03.267146 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:03.266339 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67dcc64f86-rdbqv"] Oct 30 20:44:03.267146 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:03.266566 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-67dcc64f86-rdbqv" podUID="7c9ef95b-439e-4657-ac41-b67752e7174c" containerName="manager" containerID="cri-o://d1576d5086fa15b1ec827d6583ac75854dd36019ee78fa96a75f1ca55649b817" gracePeriod=30 Oct 30 20:44:03.527898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:03.527868 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67dcc64f86-rdbqv" Oct 30 20:44:03.629377 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:03.629338 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c9ef95b-439e-4657-ac41-b67752e7174c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c9ef95b-439e-4657-ac41-b67752e7174c-metrics-certs\") pod \"7c9ef95b-439e-4657-ac41-b67752e7174c\" (UID: \"7c9ef95b-439e-4657-ac41-b67752e7174c\") " Oct 30 20:44:03.629558 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:03.629407 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c9ef95b-439e-4657-ac41-b67752e7174c-kube-api-access-9fc2v\" (UniqueName: \"kubernetes.io/projected/7c9ef95b-439e-4657-ac41-b67752e7174c-kube-api-access-9fc2v\") pod \"7c9ef95b-439e-4657-ac41-b67752e7174c\" (UID: \"7c9ef95b-439e-4657-ac41-b67752e7174c\") " Oct 30 20:44:03.629558 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:03.629431 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c9ef95b-439e-4657-ac41-b67752e7174c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c9ef95b-439e-4657-ac41-b67752e7174c-webhook-cert\") pod \"7c9ef95b-439e-4657-ac41-b67752e7174c\" (UID: \"7c9ef95b-439e-4657-ac41-b67752e7174c\") " Oct 30 20:44:03.632057 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:03.631961 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c9ef95b-439e-4657-ac41-b67752e7174c-metrics-certs" pod "7c9ef95b-439e-4657-ac41-b67752e7174c" (UID: "7c9ef95b-439e-4657-ac41-b67752e7174c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:44:03.632057 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:03.632015 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9ef95b-439e-4657-ac41-b67752e7174c-kube-api-access-9fc2v" pod "7c9ef95b-439e-4657-ac41-b67752e7174c" (UID: "7c9ef95b-439e-4657-ac41-b67752e7174c"). InnerVolumeSpecName "kube-api-access-9fc2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:44:03.632251 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:03.632051 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c9ef95b-439e-4657-ac41-b67752e7174c-webhook-cert" pod "7c9ef95b-439e-4657-ac41-b67752e7174c" (UID: "7c9ef95b-439e-4657-ac41-b67752e7174c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:44:03.730172 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:03.730131 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c9ef95b-439e-4657-ac41-b67752e7174c-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:03.730172 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:03.730173 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fc2v\" (UniqueName: \"kubernetes.io/projected/7c9ef95b-439e-4657-ac41-b67752e7174c-kube-api-access-9fc2v\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:03.730394 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:03.730190 4848 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7c9ef95b-439e-4657-ac41-b67752e7174c-webhook-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:04.251272 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:04.251239 4848 generic.go:356] "Generic (PLEG): container finished" podID="7c9ef95b-439e-4657-ac41-b67752e7174c" containerID="d1576d5086fa15b1ec827d6583ac75854dd36019ee78fa96a75f1ca55649b817" exitCode=0 Oct 30 20:44:04.251711 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:04.251281 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67dcc64f86-rdbqv" event={"ID":"7c9ef95b-439e-4657-ac41-b67752e7174c","Type":"ContainerDied","Data":"d1576d5086fa15b1ec827d6583ac75854dd36019ee78fa96a75f1ca55649b817"} Oct 30 20:44:04.251711 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:04.251292 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67dcc64f86-rdbqv" Oct 30 20:44:04.251711 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:04.251337 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67dcc64f86-rdbqv" event={"ID":"7c9ef95b-439e-4657-ac41-b67752e7174c","Type":"ContainerDied","Data":"feb097324ee168b70c447dac665076254e921dff3acead1acf881209f79e88ea"} Oct 30 20:44:04.251711 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:04.251370 4848 scope.go:122] "RemoveContainer" containerID="d1576d5086fa15b1ec827d6583ac75854dd36019ee78fa96a75f1ca55649b817" Oct 30 20:44:04.261232 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:04.261208 4848 scope.go:122] "RemoveContainer" containerID="d1576d5086fa15b1ec827d6583ac75854dd36019ee78fa96a75f1ca55649b817" Oct 30 20:44:04.262666 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:44:04.262643 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1576d5086fa15b1ec827d6583ac75854dd36019ee78fa96a75f1ca55649b817\": container with ID starting with d1576d5086fa15b1ec827d6583ac75854dd36019ee78fa96a75f1ca55649b817 not found: ID does not exist" containerID="d1576d5086fa15b1ec827d6583ac75854dd36019ee78fa96a75f1ca55649b817" Oct 30 20:44:04.262759 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:04.262676 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1576d5086fa15b1ec827d6583ac75854dd36019ee78fa96a75f1ca55649b817"} err="failed to get container status \"d1576d5086fa15b1ec827d6583ac75854dd36019ee78fa96a75f1ca55649b817\": rpc error: code = NotFound desc = could not find container \"d1576d5086fa15b1ec827d6583ac75854dd36019ee78fa96a75f1ca55649b817\": container with ID starting with d1576d5086fa15b1ec827d6583ac75854dd36019ee78fa96a75f1ca55649b817 not found: ID does not exist" Oct 30 20:44:04.281214 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:04.281187 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67dcc64f86-rdbqv"] Oct 30 20:44:04.284752 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:04.284728 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-67dcc64f86-rdbqv"] Oct 30 20:44:04.931271 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:04.931233 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c9ef95b-439e-4657-ac41-b67752e7174c" path="/var/lib/kubelet/pods/7c9ef95b-439e-4657-ac41-b67752e7174c/volumes" Oct 30 20:44:06.261676 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:06.261638 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l" event={"ID":"7dce4323-1225-4bf8-b93e-a2a482b657a5","Type":"ContainerStarted","Data":"75fa9b636197265d5dc88ef462d371f2b226e5d91b0ef34c2fc3cbfa9b3fd438"} Oct 30 20:44:06.262178 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:06.261746 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l" Oct 30 20:44:17.270297 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:17.270269 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l" Oct 30 20:44:17.294707 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:17.294662 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-rf52l" podStartSLOduration=13.383594327 podStartE2EDuration="24.294648811s" totalImagesPullingTime="10.911054484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:43:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:06.280672745 +0000 UTC m=+351.460569251" watchObservedRunningTime="2026-10-30 20:44:17.294648811 +0000 UTC m=+362.474545318" Oct 30 20:44:22.507557 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:44:22.507514 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:44:25.594333 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.594302 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7"] Oct 30 20:44:25.594744 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.594640 4848 cpu_manager.go:414] "containerMap: removing container" podUID="7c9ef95b-439e-4657-ac41-b67752e7174c" containerName="manager" Oct 30 20:44:25.594744 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.594653 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c9ef95b-439e-4657-ac41-b67752e7174c" containerName="manager" Oct 30 20:44:25.594744 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.594712 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c9ef95b-439e-4657-ac41-b67752e7174c" containerName="manager" Oct 30 20:44:25.601755 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.601736 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" Oct 30 20:44:25.610807 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.610786 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 20:44:25.610807 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.610799 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 20:44:25.627743 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.627719 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7"] Oct 30 20:44:25.705531 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.705503 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkh4s\" (UniqueName: \"kubernetes.io/projected/5740d5b9-1dd6-41bf-b362-926319a01ef1-kube-api-access-kkh4s\") pod \"maas-api-65945fbb96-6x6b7\" (UID: \"5740d5b9-1dd6-41bf-b362-926319a01ef1\") " pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" Oct 30 20:44:25.705680 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.705541 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5740d5b9-1dd6-41bf-b362-926319a01ef1-metrics-certs\") pod \"maas-api-65945fbb96-6x6b7\" (UID: \"5740d5b9-1dd6-41bf-b362-926319a01ef1\") " pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" Oct 30 20:44:25.705680 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.705561 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5740d5b9-1dd6-41bf-b362-926319a01ef1-openshift-service-ca\") pod \"maas-api-65945fbb96-6x6b7\" (UID: \"5740d5b9-1dd6-41bf-b362-926319a01ef1\") " pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" Oct 30 20:44:25.705680 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.705623 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5740d5b9-1dd6-41bf-b362-926319a01ef1-maas-api-tls\") pod \"maas-api-65945fbb96-6x6b7\" (UID: \"5740d5b9-1dd6-41bf-b362-926319a01ef1\") " pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" Oct 30 20:44:25.805994 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.805967 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkh4s\" (UniqueName: \"kubernetes.io/projected/5740d5b9-1dd6-41bf-b362-926319a01ef1-kube-api-access-kkh4s\") pod \"maas-api-65945fbb96-6x6b7\" (UID: \"5740d5b9-1dd6-41bf-b362-926319a01ef1\") " pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" Oct 30 20:44:25.806141 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.806007 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5740d5b9-1dd6-41bf-b362-926319a01ef1-metrics-certs\") pod \"maas-api-65945fbb96-6x6b7\" (UID: \"5740d5b9-1dd6-41bf-b362-926319a01ef1\") " pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" Oct 30 20:44:25.806141 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.806025 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5740d5b9-1dd6-41bf-b362-926319a01ef1-openshift-service-ca\") pod \"maas-api-65945fbb96-6x6b7\" (UID: \"5740d5b9-1dd6-41bf-b362-926319a01ef1\") " pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" Oct 30 20:44:25.806234 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.806142 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5740d5b9-1dd6-41bf-b362-926319a01ef1-maas-api-tls\") pod \"maas-api-65945fbb96-6x6b7\" (UID: \"5740d5b9-1dd6-41bf-b362-926319a01ef1\") " pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" Oct 30 20:44:25.806751 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.806731 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5740d5b9-1dd6-41bf-b362-926319a01ef1-openshift-service-ca\") pod \"maas-api-65945fbb96-6x6b7\" (UID: \"5740d5b9-1dd6-41bf-b362-926319a01ef1\") " pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" Oct 30 20:44:25.808429 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.808402 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5740d5b9-1dd6-41bf-b362-926319a01ef1-metrics-certs\") pod \"maas-api-65945fbb96-6x6b7\" (UID: \"5740d5b9-1dd6-41bf-b362-926319a01ef1\") " pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" Oct 30 20:44:25.808530 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.808503 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5740d5b9-1dd6-41bf-b362-926319a01ef1-maas-api-tls\") pod \"maas-api-65945fbb96-6x6b7\" (UID: \"5740d5b9-1dd6-41bf-b362-926319a01ef1\") " pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" Oct 30 20:44:25.819333 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.819313 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkh4s\" (UniqueName: \"kubernetes.io/projected/5740d5b9-1dd6-41bf-b362-926319a01ef1-kube-api-access-kkh4s\") pod \"maas-api-65945fbb96-6x6b7\" (UID: \"5740d5b9-1dd6-41bf-b362-926319a01ef1\") " pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" Oct 30 20:44:25.915240 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:25.915170 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" Oct 30 20:44:26.189615 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:26.189589 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7"] Oct 30 20:44:26.192099 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:44:26.192062 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5740d5b9_1dd6_41bf_b362_926319a01ef1.slice/crio-ed233be1808de341a4701415f7af9e0705635bdd2ec3f0c125b8f3e0d3ea8c9a WatchSource:0}: Error finding container ed233be1808de341a4701415f7af9e0705635bdd2ec3f0c125b8f3e0d3ea8c9a: Status 404 returned error can't find the container with id ed233be1808de341a4701415f7af9e0705635bdd2ec3f0c125b8f3e0d3ea8c9a Oct 30 20:44:26.338070 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:26.338030 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" event={"ID":"5740d5b9-1dd6-41bf-b362-926319a01ef1","Type":"ContainerStarted","Data":"ed233be1808de341a4701415f7af9e0705635bdd2ec3f0c125b8f3e0d3ea8c9a"} Oct 30 20:44:30.356008 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:30.355964 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" event={"ID":"5740d5b9-1dd6-41bf-b362-926319a01ef1","Type":"ContainerStarted","Data":"9da9fb9857d7de41ed93581aa8641b9e23a1aa60bcf992f65e0aab4e8090200f"} Oct 30 20:44:30.356451 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:30.356159 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" Oct 30 20:44:30.378152 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:30.378099 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" podStartSLOduration=1.675141577 podStartE2EDuration="5.378062968s" totalImagesPullingTime="3.702921391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:30.377257822 +0000 UTC m=+375.557154329" watchObservedRunningTime="2026-10-30 20:44:30.378062968 +0000 UTC m=+375.557959476" Oct 30 20:44:36.365890 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:36.365864 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" Oct 30 20:44:37.082708 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.082674 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dd76875b5-8p6dg"] Oct 30 20:44:37.368785 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.368684 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-8p6dg"] Oct 30 20:44:37.368785 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.368729 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-8p6dg"] Oct 30 20:44:37.368785 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.368747 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2pqvr"] Oct 30 20:44:37.369227 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.368810 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-8p6dg" Oct 30 20:44:37.369316 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:44:37.369293 4848 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-v2z8z tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-v2z8z tls-cert]: context canceled" pod="kuadrant-system/authorino-6dd76875b5-8p6dg" podUID="df81f4d3-7c3e-4ff5-a239-2181941d66e5" Oct 30 20:44:37.374297 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.374281 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2pqvr"] Oct 30 20:44:37.374371 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.374353 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2pqvr" Oct 30 20:44:37.386657 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.386637 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-8p6dg" Oct 30 20:44:37.399707 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.399606 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-8p6dg" Oct 30 20:44:37.406086 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.406070 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 20:44:37.471804 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.471756 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2pqvr"] Oct 30 20:44:37.471984 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:44:37.471965 4848 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-bcm6n tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-2pqvr" podUID="881bafd2-1b41-4d81-a78f-2796668a4bd1" Oct 30 20:44:37.500315 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.500286 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-798dd5cc85-s2vpq"] Oct 30 20:44:37.512577 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.512546 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/881bafd2-1b41-4d81-a78f-2796668a4bd1-tls-cert\") pod \"authorino-6889cbd8c4-2pqvr\" (UID: \"881bafd2-1b41-4d81-a78f-2796668a4bd1\") " pod="kuadrant-system/authorino-6889cbd8c4-2pqvr" Oct 30 20:44:37.512725 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.512630 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcm6n\" (UniqueName: \"kubernetes.io/projected/881bafd2-1b41-4d81-a78f-2796668a4bd1-kube-api-access-bcm6n\") pod \"authorino-6889cbd8c4-2pqvr\" (UID: \"881bafd2-1b41-4d81-a78f-2796668a4bd1\") " pod="kuadrant-system/authorino-6889cbd8c4-2pqvr" Oct 30 20:44:37.527848 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.527821 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-798dd5cc85-s2vpq"] Oct 30 20:44:37.527956 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.527910 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-798dd5cc85-s2vpq" Oct 30 20:44:37.613660 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.613623 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcm6n\" (UniqueName: \"kubernetes.io/projected/881bafd2-1b41-4d81-a78f-2796668a4bd1-kube-api-access-bcm6n\") pod \"authorino-6889cbd8c4-2pqvr\" (UID: \"881bafd2-1b41-4d81-a78f-2796668a4bd1\") " pod="kuadrant-system/authorino-6889cbd8c4-2pqvr" Oct 30 20:44:37.613854 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.613679 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jph6m\" (UniqueName: \"kubernetes.io/projected/54b58047-e429-408e-9376-9697f523b381-kube-api-access-jph6m\") pod \"authorino-798dd5cc85-s2vpq\" (UID: \"54b58047-e429-408e-9376-9697f523b381\") " pod="kuadrant-system/authorino-798dd5cc85-s2vpq" Oct 30 20:44:37.613854 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.613708 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/54b58047-e429-408e-9376-9697f523b381-tls-cert\") pod \"authorino-798dd5cc85-s2vpq\" (UID: \"54b58047-e429-408e-9376-9697f523b381\") " pod="kuadrant-system/authorino-798dd5cc85-s2vpq" Oct 30 20:44:37.613938 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.613853 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/881bafd2-1b41-4d81-a78f-2796668a4bd1-tls-cert\") pod \"authorino-6889cbd8c4-2pqvr\" (UID: \"881bafd2-1b41-4d81-a78f-2796668a4bd1\") " pod="kuadrant-system/authorino-6889cbd8c4-2pqvr" Oct 30 20:44:37.616241 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.616214 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/881bafd2-1b41-4d81-a78f-2796668a4bd1-tls-cert\") pod \"authorino-6889cbd8c4-2pqvr\" (UID: \"881bafd2-1b41-4d81-a78f-2796668a4bd1\") " pod="kuadrant-system/authorino-6889cbd8c4-2pqvr" Oct 30 20:44:37.622966 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.622919 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcm6n\" (UniqueName: \"kubernetes.io/projected/881bafd2-1b41-4d81-a78f-2796668a4bd1-kube-api-access-bcm6n\") pod \"authorino-6889cbd8c4-2pqvr\" (UID: \"881bafd2-1b41-4d81-a78f-2796668a4bd1\") " pod="kuadrant-system/authorino-6889cbd8c4-2pqvr" Oct 30 20:44:37.715160 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.715123 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/54b58047-e429-408e-9376-9697f523b381-tls-cert\") pod \"authorino-798dd5cc85-s2vpq\" (UID: \"54b58047-e429-408e-9376-9697f523b381\") " pod="kuadrant-system/authorino-798dd5cc85-s2vpq" Oct 30 20:44:37.715290 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.715210 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jph6m\" (UniqueName: \"kubernetes.io/projected/54b58047-e429-408e-9376-9697f523b381-kube-api-access-jph6m\") pod \"authorino-798dd5cc85-s2vpq\" (UID: \"54b58047-e429-408e-9376-9697f523b381\") " pod="kuadrant-system/authorino-798dd5cc85-s2vpq" Oct 30 20:44:37.717633 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.717610 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/54b58047-e429-408e-9376-9697f523b381-tls-cert\") pod \"authorino-798dd5cc85-s2vpq\" (UID: \"54b58047-e429-408e-9376-9697f523b381\") " pod="kuadrant-system/authorino-798dd5cc85-s2vpq" Oct 30 20:44:37.724149 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.724126 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jph6m\" (UniqueName: \"kubernetes.io/projected/54b58047-e429-408e-9376-9697f523b381-kube-api-access-jph6m\") pod \"authorino-798dd5cc85-s2vpq\" (UID: \"54b58047-e429-408e-9376-9697f523b381\") " pod="kuadrant-system/authorino-798dd5cc85-s2vpq" Oct 30 20:44:37.844746 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:37.844713 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-798dd5cc85-s2vpq" Oct 30 20:44:38.159081 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:38.159058 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-798dd5cc85-s2vpq"] Oct 30 20:44:38.161617 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:44:38.161588 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54b58047_e429_408e_9376_9697f523b381.slice/crio-938bd8bb82aa83d62f218a9fe773cf1b0af29cbacb1abe51f30fec2c29085332 WatchSource:0}: Error finding container 938bd8bb82aa83d62f218a9fe773cf1b0af29cbacb1abe51f30fec2c29085332: Status 404 returned error can't find the container with id 938bd8bb82aa83d62f218a9fe773cf1b0af29cbacb1abe51f30fec2c29085332 Oct 30 20:44:38.392199 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:38.392162 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-798dd5cc85-s2vpq" event={"ID":"54b58047-e429-408e-9376-9697f523b381","Type":"ContainerStarted","Data":"938bd8bb82aa83d62f218a9fe773cf1b0af29cbacb1abe51f30fec2c29085332"} Oct 30 20:44:38.392199 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:38.392182 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2pqvr" Oct 30 20:44:38.392682 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:38.392184 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-8p6dg" Oct 30 20:44:38.411964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:38.411909 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2pqvr" Oct 30 20:44:38.447113 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:38.447087 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-8p6dg"] Oct 30 20:44:38.452126 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:38.452105 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-8p6dg"] Oct 30 20:44:38.520956 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:38.520929 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/881bafd2-1b41-4d81-a78f-2796668a4bd1-kube-api-access-bcm6n\" (UniqueName: \"kubernetes.io/projected/881bafd2-1b41-4d81-a78f-2796668a4bd1-kube-api-access-bcm6n\") pod \"881bafd2-1b41-4d81-a78f-2796668a4bd1\" (UID: \"881bafd2-1b41-4d81-a78f-2796668a4bd1\") " Oct 30 20:44:38.521086 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:38.520976 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/881bafd2-1b41-4d81-a78f-2796668a4bd1-tls-cert\" (UniqueName: \"kubernetes.io/secret/881bafd2-1b41-4d81-a78f-2796668a4bd1-tls-cert\") pod \"881bafd2-1b41-4d81-a78f-2796668a4bd1\" (UID: \"881bafd2-1b41-4d81-a78f-2796668a4bd1\") " Oct 30 20:44:38.522991 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:38.522964 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/881bafd2-1b41-4d81-a78f-2796668a4bd1-tls-cert" pod "881bafd2-1b41-4d81-a78f-2796668a4bd1" (UID: "881bafd2-1b41-4d81-a78f-2796668a4bd1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:44:38.523086 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:38.523003 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881bafd2-1b41-4d81-a78f-2796668a4bd1-kube-api-access-bcm6n" pod "881bafd2-1b41-4d81-a78f-2796668a4bd1" (UID: "881bafd2-1b41-4d81-a78f-2796668a4bd1"). InnerVolumeSpecName "kube-api-access-bcm6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:44:38.621518 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:38.621457 4848 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/881bafd2-1b41-4d81-a78f-2796668a4bd1-tls-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:38.621518 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:38.621482 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcm6n\" (UniqueName: \"kubernetes.io/projected/881bafd2-1b41-4d81-a78f-2796668a4bd1-kube-api-access-bcm6n\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:38.929721 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:38.929688 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df81f4d3-7c3e-4ff5-a239-2181941d66e5" path="/var/lib/kubelet/pods/df81f4d3-7c3e-4ff5-a239-2181941d66e5/volumes" Oct 30 20:44:39.398260 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:39.398233 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-798dd5cc85-s2vpq" event={"ID":"54b58047-e429-408e-9376-9697f523b381","Type":"ContainerStarted","Data":"ad66066ab381e0e15549b2d83539adf4642e5f400d40e6f6c18e1bece4600675"} Oct 30 20:44:39.398653 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:39.398372 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2pqvr" Oct 30 20:44:39.417241 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:39.417199 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-798dd5cc85-s2vpq" podStartSLOduration=1.246879621 podStartE2EDuration="2.417185682s" totalImagesPullingTime="1.170306061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:44:39.415619905 +0000 UTC m=+384.595516411" watchObservedRunningTime="2026-10-30 20:44:39.417185682 +0000 UTC m=+384.597082192" Oct 30 20:44:39.445410 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:39.445373 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2pqvr"] Oct 30 20:44:39.451686 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:39.451660 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zpnjg"] Oct 30 20:44:39.452199 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:39.452030 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-zpnjg" podUID="087322b1-be2a-4aea-a31d-3e5d2fcd2340" containerName="authorino" containerID="cri-o://ed26988b2adc7c099315703a87e40f9f1d5adc5e2c1673907486f237d1732197" gracePeriod=30 Oct 30 20:44:39.453073 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:39.453050 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2pqvr"] Oct 30 20:44:39.706055 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:39.706030 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zpnjg" Oct 30 20:44:39.832254 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:39.832216 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/087322b1-be2a-4aea-a31d-3e5d2fcd2340-kube-api-access-74lct\" (UniqueName: \"kubernetes.io/projected/087322b1-be2a-4aea-a31d-3e5d2fcd2340-kube-api-access-74lct\") pod \"087322b1-be2a-4aea-a31d-3e5d2fcd2340\" (UID: \"087322b1-be2a-4aea-a31d-3e5d2fcd2340\") " Oct 30 20:44:39.834388 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:39.834356 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087322b1-be2a-4aea-a31d-3e5d2fcd2340-kube-api-access-74lct" pod "087322b1-be2a-4aea-a31d-3e5d2fcd2340" (UID: "087322b1-be2a-4aea-a31d-3e5d2fcd2340"). InnerVolumeSpecName "kube-api-access-74lct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:44:39.937297 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:39.937259 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74lct\" (UniqueName: \"kubernetes.io/projected/087322b1-be2a-4aea-a31d-3e5d2fcd2340-kube-api-access-74lct\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:40.404194 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:40.404117 4848 generic.go:356] "Generic (PLEG): container finished" podID="087322b1-be2a-4aea-a31d-3e5d2fcd2340" containerID="ed26988b2adc7c099315703a87e40f9f1d5adc5e2c1673907486f237d1732197" exitCode=0 Oct 30 20:44:40.404194 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:40.404162 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zpnjg" Oct 30 20:44:40.404623 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:40.404196 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zpnjg" event={"ID":"087322b1-be2a-4aea-a31d-3e5d2fcd2340","Type":"ContainerDied","Data":"ed26988b2adc7c099315703a87e40f9f1d5adc5e2c1673907486f237d1732197"} Oct 30 20:44:40.404623 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:40.404239 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zpnjg" event={"ID":"087322b1-be2a-4aea-a31d-3e5d2fcd2340","Type":"ContainerDied","Data":"647e9bef18316299fcffdd02a85b73293f0a81bb81484347ee94986ff0ba60b4"} Oct 30 20:44:40.404623 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:40.404257 4848 scope.go:122] "RemoveContainer" containerID="ed26988b2adc7c099315703a87e40f9f1d5adc5e2c1673907486f237d1732197" Oct 30 20:44:40.413910 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:40.413790 4848 scope.go:122] "RemoveContainer" containerID="ed26988b2adc7c099315703a87e40f9f1d5adc5e2c1673907486f237d1732197" Oct 30 20:44:40.414649 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:44:40.414620 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed26988b2adc7c099315703a87e40f9f1d5adc5e2c1673907486f237d1732197\": container with ID starting with ed26988b2adc7c099315703a87e40f9f1d5adc5e2c1673907486f237d1732197 not found: ID does not exist" containerID="ed26988b2adc7c099315703a87e40f9f1d5adc5e2c1673907486f237d1732197" Oct 30 20:44:40.414745 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:40.414659 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed26988b2adc7c099315703a87e40f9f1d5adc5e2c1673907486f237d1732197"} err="failed to get container status \"ed26988b2adc7c099315703a87e40f9f1d5adc5e2c1673907486f237d1732197\": rpc error: code = NotFound desc = could not find container \"ed26988b2adc7c099315703a87e40f9f1d5adc5e2c1673907486f237d1732197\": container with ID starting with ed26988b2adc7c099315703a87e40f9f1d5adc5e2c1673907486f237d1732197 not found: ID does not exist" Oct 30 20:44:40.441557 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:40.441535 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zpnjg"] Oct 30 20:44:40.446329 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:40.446311 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zpnjg"] Oct 30 20:44:40.929734 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:40.929701 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="087322b1-be2a-4aea-a31d-3e5d2fcd2340" path="/var/lib/kubelet/pods/087322b1-be2a-4aea-a31d-3e5d2fcd2340/volumes" Oct 30 20:44:40.930106 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:40.930091 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="881bafd2-1b41-4d81-a78f-2796668a4bd1" path="/var/lib/kubelet/pods/881bafd2-1b41-4d81-a78f-2796668a4bd1/volumes" Oct 30 20:44:47.245024 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:47.244946 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7"] Oct 30 20:44:47.245491 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:47.245168 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" podUID="5740d5b9-1dd6-41bf-b362-926319a01ef1" containerName="maas-api" containerID="cri-o://9da9fb9857d7de41ed93581aa8641b9e23a1aa60bcf992f65e0aab4e8090200f" gracePeriod=30 Oct 30 20:44:47.433648 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:47.433614 4848 generic.go:356] "Generic (PLEG): container finished" podID="5740d5b9-1dd6-41bf-b362-926319a01ef1" containerID="9da9fb9857d7de41ed93581aa8641b9e23a1aa60bcf992f65e0aab4e8090200f" exitCode=0 Oct 30 20:44:47.433836 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:47.433685 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" event={"ID":"5740d5b9-1dd6-41bf-b362-926319a01ef1","Type":"ContainerDied","Data":"9da9fb9857d7de41ed93581aa8641b9e23a1aa60bcf992f65e0aab4e8090200f"} Oct 30 20:44:47.486336 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:47.486305 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" Oct 30 20:44:47.596243 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:47.596163 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5740d5b9-1dd6-41bf-b362-926319a01ef1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5740d5b9-1dd6-41bf-b362-926319a01ef1-maas-api-tls\") pod \"5740d5b9-1dd6-41bf-b362-926319a01ef1\" (UID: \"5740d5b9-1dd6-41bf-b362-926319a01ef1\") " Oct 30 20:44:47.596243 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:47.596213 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5740d5b9-1dd6-41bf-b362-926319a01ef1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5740d5b9-1dd6-41bf-b362-926319a01ef1-metrics-certs\") pod \"5740d5b9-1dd6-41bf-b362-926319a01ef1\" (UID: \"5740d5b9-1dd6-41bf-b362-926319a01ef1\") " Oct 30 20:44:47.596243 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:47.596237 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5740d5b9-1dd6-41bf-b362-926319a01ef1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5740d5b9-1dd6-41bf-b362-926319a01ef1-openshift-service-ca\") pod \"5740d5b9-1dd6-41bf-b362-926319a01ef1\" (UID: \"5740d5b9-1dd6-41bf-b362-926319a01ef1\") " Oct 30 20:44:47.596498 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:47.596340 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5740d5b9-1dd6-41bf-b362-926319a01ef1-kube-api-access-kkh4s\" (UniqueName: \"kubernetes.io/projected/5740d5b9-1dd6-41bf-b362-926319a01ef1-kube-api-access-kkh4s\") pod \"5740d5b9-1dd6-41bf-b362-926319a01ef1\" (UID: \"5740d5b9-1dd6-41bf-b362-926319a01ef1\") " Oct 30 20:44:47.596752 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:47.596725 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5740d5b9-1dd6-41bf-b362-926319a01ef1-openshift-service-ca" pod "5740d5b9-1dd6-41bf-b362-926319a01ef1" (UID: "5740d5b9-1dd6-41bf-b362-926319a01ef1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:44:47.598455 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:47.598422 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5740d5b9-1dd6-41bf-b362-926319a01ef1-metrics-certs" pod "5740d5b9-1dd6-41bf-b362-926319a01ef1" (UID: "5740d5b9-1dd6-41bf-b362-926319a01ef1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:44:47.598545 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:47.598481 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5740d5b9-1dd6-41bf-b362-926319a01ef1-maas-api-tls" pod "5740d5b9-1dd6-41bf-b362-926319a01ef1" (UID: "5740d5b9-1dd6-41bf-b362-926319a01ef1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:44:47.601264 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:47.601238 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5740d5b9-1dd6-41bf-b362-926319a01ef1-kube-api-access-kkh4s" pod "5740d5b9-1dd6-41bf-b362-926319a01ef1" (UID: "5740d5b9-1dd6-41bf-b362-926319a01ef1"). InnerVolumeSpecName "kube-api-access-kkh4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:44:47.697125 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:47.697088 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5740d5b9-1dd6-41bf-b362-926319a01ef1-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:47.697125 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:47.697119 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5740d5b9-1dd6-41bf-b362-926319a01ef1-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:47.697125 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:47.697130 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5740d5b9-1dd6-41bf-b362-926319a01ef1-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:47.697356 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:47.697139 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkh4s\" (UniqueName: \"kubernetes.io/projected/5740d5b9-1dd6-41bf-b362-926319a01ef1-kube-api-access-kkh4s\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:44:48.439636 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:48.439600 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" event={"ID":"5740d5b9-1dd6-41bf-b362-926319a01ef1","Type":"ContainerDied","Data":"ed233be1808de341a4701415f7af9e0705635bdd2ec3f0c125b8f3e0d3ea8c9a"} Oct 30 20:44:48.440074 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:48.439657 4848 scope.go:122] "RemoveContainer" containerID="9da9fb9857d7de41ed93581aa8641b9e23a1aa60bcf992f65e0aab4e8090200f" Oct 30 20:44:48.440074 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:48.439614 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7" Oct 30 20:44:48.461586 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:48.461564 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7"] Oct 30 20:44:48.466813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:48.466793 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-65945fbb96-6x6b7"] Oct 30 20:44:48.929426 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:44:48.929395 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5740d5b9-1dd6-41bf-b362-926319a01ef1" path="/var/lib/kubelet/pods/5740d5b9-1dd6-41bf-b362-926319a01ef1/volumes" Oct 30 20:44:52.509784 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:44:52.509726 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:45:00.148692 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:00.148661 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb"] Oct 30 20:45:00.149078 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:00.148981 4848 cpu_manager.go:414] "containerMap: removing container" podUID="5740d5b9-1dd6-41bf-b362-926319a01ef1" containerName="maas-api" Oct 30 20:45:00.149078 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:00.148992 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5740d5b9-1dd6-41bf-b362-926319a01ef1" containerName="maas-api" Oct 30 20:45:00.149078 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:00.149015 4848 cpu_manager.go:414] "containerMap: removing container" podUID="087322b1-be2a-4aea-a31d-3e5d2fcd2340" containerName="authorino" Oct 30 20:45:00.149078 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:00.149020 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="087322b1-be2a-4aea-a31d-3e5d2fcd2340" containerName="authorino" Oct 30 20:45:00.149078 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:00.149062 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="087322b1-be2a-4aea-a31d-3e5d2fcd2340" containerName="authorino" Oct 30 20:45:00.149078 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:00.149072 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="5740d5b9-1dd6-41bf-b362-926319a01ef1" containerName="maas-api" Oct 30 20:45:00.153861 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:00.153839 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" Oct 30 20:45:00.159937 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:00.159914 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb"] Oct 30 20:45:00.160391 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:00.160367 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\"" Oct 30 20:45:00.198820 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:00.198788 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84cxm\" (UniqueName: \"kubernetes.io/projected/577190e1-ed57-4e93-9b81-e18111117ad0-kube-api-access-84cxm\") pod \"maas-api-key-cleanup-29889885-g6prb\" (UID: \"577190e1-ed57-4e93-9b81-e18111117ad0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" Oct 30 20:45:00.198938 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:00.198896 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/577190e1-ed57-4e93-9b81-e18111117ad0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889885-g6prb\" (UID: \"577190e1-ed57-4e93-9b81-e18111117ad0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" Oct 30 20:45:00.300053 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:00.300024 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84cxm\" (UniqueName: \"kubernetes.io/projected/577190e1-ed57-4e93-9b81-e18111117ad0-kube-api-access-84cxm\") pod \"maas-api-key-cleanup-29889885-g6prb\" (UID: \"577190e1-ed57-4e93-9b81-e18111117ad0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" Oct 30 20:45:00.300160 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:00.300076 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/577190e1-ed57-4e93-9b81-e18111117ad0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889885-g6prb\" (UID: \"577190e1-ed57-4e93-9b81-e18111117ad0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" Oct 30 20:45:00.300616 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:00.300598 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/577190e1-ed57-4e93-9b81-e18111117ad0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889885-g6prb\" (UID: \"577190e1-ed57-4e93-9b81-e18111117ad0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" Oct 30 20:45:00.310285 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:00.310253 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84cxm\" (UniqueName: \"kubernetes.io/projected/577190e1-ed57-4e93-9b81-e18111117ad0-kube-api-access-84cxm\") pod \"maas-api-key-cleanup-29889885-g6prb\" (UID: \"577190e1-ed57-4e93-9b81-e18111117ad0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" Oct 30 20:45:00.468549 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:00.468525 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" Oct 30 20:45:00.740525 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:00.740501 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb"] Oct 30 20:45:00.742878 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:45:00.742850 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577190e1_ed57_4e93_9b81_e18111117ad0.slice/crio-79dd30773062a8189a7db6b2436c2c48778c5b62e1b2b86e8f997fd14e37b14d WatchSource:0}: Error finding container 79dd30773062a8189a7db6b2436c2c48778c5b62e1b2b86e8f997fd14e37b14d: Status 404 returned error can't find the container with id 79dd30773062a8189a7db6b2436c2c48778c5b62e1b2b86e8f997fd14e37b14d Oct 30 20:45:01.493488 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:01.493452 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" event={"ID":"577190e1-ed57-4e93-9b81-e18111117ad0","Type":"ContainerStarted","Data":"79dd30773062a8189a7db6b2436c2c48778c5b62e1b2b86e8f997fd14e37b14d"} Oct 30 20:45:03.504606 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:03.504573 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" event={"ID":"577190e1-ed57-4e93-9b81-e18111117ad0","Type":"ContainerStarted","Data":"af908c87104161315c2bc3250495f9686c0fec6b09d33332d3dec67486b3ddd4"} Oct 30 20:45:03.524292 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:03.524251 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" podStartSLOduration=1.195951617 podStartE2EDuration="3.524240562s" totalImagesPullingTime="2.328288945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:03.522683508 +0000 UTC m=+408.702580014" watchObservedRunningTime="2026-10-30 20:45:03.524240562 +0000 UTC m=+408.704137099" Oct 30 20:45:04.509916 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:04.509879 4848 generic.go:356] "Generic (PLEG): container finished" podID="577190e1-ed57-4e93-9b81-e18111117ad0" containerID="af908c87104161315c2bc3250495f9686c0fec6b09d33332d3dec67486b3ddd4" exitCode=60 Oct 30 20:45:04.510364 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:04.509963 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" event={"ID":"577190e1-ed57-4e93-9b81-e18111117ad0","Type":"ContainerDied","Data":"af908c87104161315c2bc3250495f9686c0fec6b09d33332d3dec67486b3ddd4"} Oct 30 20:45:04.510364 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:04.510142 4848 scope.go:122] "RemoveContainer" containerID="af908c87104161315c2bc3250495f9686c0fec6b09d33332d3dec67486b3ddd4" Oct 30 20:45:05.515439 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:05.515405 4848 generic.go:356] "Generic (PLEG): container finished" podID="577190e1-ed57-4e93-9b81-e18111117ad0" containerID="d7e885f2751723e103833d947dee52538915e63061c58820fe2d38e521881457" exitCode=60 Oct 30 20:45:05.515897 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:05.515442 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" event={"ID":"577190e1-ed57-4e93-9b81-e18111117ad0","Type":"ContainerDied","Data":"d7e885f2751723e103833d947dee52538915e63061c58820fe2d38e521881457"} Oct 30 20:45:05.515897 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:05.515506 4848 scope.go:122] "RemoveContainer" containerID="af908c87104161315c2bc3250495f9686c0fec6b09d33332d3dec67486b3ddd4" Oct 30 20:45:05.515897 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:05.515711 4848 scope.go:122] "RemoveContainer" containerID="d7e885f2751723e103833d947dee52538915e63061c58820fe2d38e521881457" Oct 30 20:45:05.516065 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:45:05.515955 4848 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889885-g6prb_odh-ai-gateway-infra(577190e1-ed57-4e93-9b81-e18111117ad0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" podUID="577190e1-ed57-4e93-9b81-e18111117ad0" Oct 30 20:45:06.521564 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:06.521534 4848 scope.go:122] "RemoveContainer" containerID="d7e885f2751723e103833d947dee52538915e63061c58820fe2d38e521881457" Oct 30 20:45:06.521951 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:45:06.521733 4848 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889885-g6prb_odh-ai-gateway-infra(577190e1-ed57-4e93-9b81-e18111117ad0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" podUID="577190e1-ed57-4e93-9b81-e18111117ad0" Oct 30 20:45:14.828225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:14.828178 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-m2n56"] Oct 30 20:45:14.831387 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:14.831368 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-m2n56" Oct 30 20:45:14.836157 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:14.836131 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-9jb48\"" Oct 30 20:45:14.836157 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:14.836146 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 20:45:14.836315 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:14.836167 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 20:45:14.840160 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:14.840136 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-m2n56"] Oct 30 20:45:14.907313 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:14.907278 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e57fcd7a-e48b-4935-96ea-0e8bf7d3258b-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-m2n56\" (UID: \"e57fcd7a-e48b-4935-96ea-0e8bf7d3258b\") " pod="opendatahub/ai-gateway-controller-7d85675f98-m2n56" Oct 30 20:45:14.907313 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:14.907317 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng468\" (UniqueName: \"kubernetes.io/projected/e57fcd7a-e48b-4935-96ea-0e8bf7d3258b-kube-api-access-ng468\") pod \"ai-gateway-controller-7d85675f98-m2n56\" (UID: \"e57fcd7a-e48b-4935-96ea-0e8bf7d3258b\") " pod="opendatahub/ai-gateway-controller-7d85675f98-m2n56" Oct 30 20:45:15.008682 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:15.008654 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e57fcd7a-e48b-4935-96ea-0e8bf7d3258b-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-m2n56\" (UID: \"e57fcd7a-e48b-4935-96ea-0e8bf7d3258b\") " pod="opendatahub/ai-gateway-controller-7d85675f98-m2n56" Oct 30 20:45:15.008822 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:15.008686 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng468\" (UniqueName: \"kubernetes.io/projected/e57fcd7a-e48b-4935-96ea-0e8bf7d3258b-kube-api-access-ng468\") pod \"ai-gateway-controller-7d85675f98-m2n56\" (UID: \"e57fcd7a-e48b-4935-96ea-0e8bf7d3258b\") " pod="opendatahub/ai-gateway-controller-7d85675f98-m2n56" Oct 30 20:45:15.011251 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:15.011228 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 20:45:15.017489 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:15.017467 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng468\" (UniqueName: \"kubernetes.io/projected/e57fcd7a-e48b-4935-96ea-0e8bf7d3258b-kube-api-access-ng468\") pod \"ai-gateway-controller-7d85675f98-m2n56\" (UID: \"e57fcd7a-e48b-4935-96ea-0e8bf7d3258b\") " pod="opendatahub/ai-gateway-controller-7d85675f98-m2n56" Oct 30 20:45:15.021117 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:15.021095 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e57fcd7a-e48b-4935-96ea-0e8bf7d3258b-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-m2n56\" (UID: \"e57fcd7a-e48b-4935-96ea-0e8bf7d3258b\") " pod="opendatahub/ai-gateway-controller-7d85675f98-m2n56" Oct 30 20:45:15.146987 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:15.146904 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-9jb48\"" Oct 30 20:45:15.155030 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:15.155008 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-m2n56" Oct 30 20:45:15.425987 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:15.425961 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-m2n56"] Oct 30 20:45:15.428376 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:45:15.428345 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57fcd7a_e48b_4935_96ea_0e8bf7d3258b.slice/crio-15f80510c6bb55c7e9bc4552e2a1dc8a57eca665b060677c1dfc3a23eba2b69c WatchSource:0}: Error finding container 15f80510c6bb55c7e9bc4552e2a1dc8a57eca665b060677c1dfc3a23eba2b69c: Status 404 returned error can't find the container with id 15f80510c6bb55c7e9bc4552e2a1dc8a57eca665b060677c1dfc3a23eba2b69c Oct 30 20:45:15.557794 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:15.557748 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-m2n56" event={"ID":"e57fcd7a-e48b-4935-96ea-0e8bf7d3258b","Type":"ContainerStarted","Data":"15f80510c6bb55c7e9bc4552e2a1dc8a57eca665b060677c1dfc3a23eba2b69c"} Oct 30 20:45:18.926932 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:18.926899 4848 scope.go:122] "RemoveContainer" containerID="d7e885f2751723e103833d947dee52538915e63061c58820fe2d38e521881457" Oct 30 20:45:19.102204 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:19.102185 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 20:45:19.575284 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:19.575203 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" event={"ID":"577190e1-ed57-4e93-9b81-e18111117ad0","Type":"ContainerStarted","Data":"aa13dd8e827e6308421f8aadd45b04c7bcda147e51e89f51e059dd10334f2194"} Oct 30 20:45:19.576553 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:19.576529 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-m2n56" event={"ID":"e57fcd7a-e48b-4935-96ea-0e8bf7d3258b","Type":"ContainerStarted","Data":"402578d1a5314529cca4c0ead61678bf120870d4b074ee132bb22b6a2f9bae36"} Oct 30 20:45:19.576679 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:19.576665 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-m2n56" Oct 30 20:45:19.627172 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:19.627108 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-m2n56" podStartSLOduration=1.957736921 podStartE2EDuration="5.627089311s" totalImagesPullingTime="3.66935239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:19.625898903 +0000 UTC m=+424.805795411" watchObservedRunningTime="2026-10-30 20:45:19.627089311 +0000 UTC m=+424.806985818" Oct 30 20:45:20.582345 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:20.582312 4848 generic.go:356] "Generic (PLEG): container finished" podID="577190e1-ed57-4e93-9b81-e18111117ad0" containerID="aa13dd8e827e6308421f8aadd45b04c7bcda147e51e89f51e059dd10334f2194" exitCode=60 Oct 30 20:45:20.582812 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:20.582391 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" event={"ID":"577190e1-ed57-4e93-9b81-e18111117ad0","Type":"ContainerDied","Data":"aa13dd8e827e6308421f8aadd45b04c7bcda147e51e89f51e059dd10334f2194"} Oct 30 20:45:20.582812 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:20.582450 4848 scope.go:122] "RemoveContainer" containerID="d7e885f2751723e103833d947dee52538915e63061c58820fe2d38e521881457" Oct 30 20:45:20.582952 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:20.582843 4848 scope.go:122] "RemoveContainer" containerID="aa13dd8e827e6308421f8aadd45b04c7bcda147e51e89f51e059dd10334f2194" Oct 30 20:45:20.583081 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:45:20.583062 4848 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889885-g6prb_odh-ai-gateway-infra(577190e1-ed57-4e93-9b81-e18111117ad0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" podUID="577190e1-ed57-4e93-9b81-e18111117ad0" Oct 30 20:45:20.617471 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:20.617436 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb"] Oct 30 20:45:20.989236 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:20.989200 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fztgr"] Oct 30 20:45:20.993727 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:20.993710 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-fztgr" Oct 30 20:45:21.002867 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.002839 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fztgr"] Oct 30 20:45:21.002999 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.002989 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 20:45:21.026399 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.026368 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-shfhk"] Oct 30 20:45:21.029794 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.029753 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-shfhk" Oct 30 20:45:21.039675 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.039651 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5ggdv\"" Oct 30 20:45:21.041869 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.041842 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-shfhk"] Oct 30 20:45:21.060757 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.060725 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce11e0b-618d-40b0-9709-d6b3deda2012-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-fztgr\" (UID: \"bce11e0b-618d-40b0-9709-d6b3deda2012\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fztgr" Oct 30 20:45:21.060957 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.060853 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7711661-bd4e-49ba-96f6-2739374b633e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-shfhk\" (UID: \"c7711661-bd4e-49ba-96f6-2739374b633e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-shfhk" Oct 30 20:45:21.060957 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.060896 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f69fh\" (UniqueName: \"kubernetes.io/projected/c7711661-bd4e-49ba-96f6-2739374b633e-kube-api-access-f69fh\") pod \"payload-pre-processing-7b98bb678c-shfhk\" (UID: \"c7711661-bd4e-49ba-96f6-2739374b633e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-shfhk" Oct 30 20:45:21.061124 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.060979 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4hh6\" (UniqueName: \"kubernetes.io/projected/bce11e0b-618d-40b0-9709-d6b3deda2012-kube-api-access-j4hh6\") pod \"payload-processing-55fbbbf9dc-fztgr\" (UID: \"bce11e0b-618d-40b0-9709-d6b3deda2012\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fztgr" Oct 30 20:45:21.162119 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.162084 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4hh6\" (UniqueName: \"kubernetes.io/projected/bce11e0b-618d-40b0-9709-d6b3deda2012-kube-api-access-j4hh6\") pod \"payload-processing-55fbbbf9dc-fztgr\" (UID: \"bce11e0b-618d-40b0-9709-d6b3deda2012\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fztgr" Oct 30 20:45:21.162296 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.162149 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce11e0b-618d-40b0-9709-d6b3deda2012-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-fztgr\" (UID: \"bce11e0b-618d-40b0-9709-d6b3deda2012\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fztgr" Oct 30 20:45:21.162296 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.162202 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7711661-bd4e-49ba-96f6-2739374b633e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-shfhk\" (UID: \"c7711661-bd4e-49ba-96f6-2739374b633e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-shfhk" Oct 30 20:45:21.162296 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.162235 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f69fh\" (UniqueName: \"kubernetes.io/projected/c7711661-bd4e-49ba-96f6-2739374b633e-kube-api-access-f69fh\") pod \"payload-pre-processing-7b98bb678c-shfhk\" (UID: \"c7711661-bd4e-49ba-96f6-2739374b633e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-shfhk" Oct 30 20:45:21.163029 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.163003 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7711661-bd4e-49ba-96f6-2739374b633e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-shfhk\" (UID: \"c7711661-bd4e-49ba-96f6-2739374b633e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-shfhk" Oct 30 20:45:21.163029 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.163017 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce11e0b-618d-40b0-9709-d6b3deda2012-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-fztgr\" (UID: \"bce11e0b-618d-40b0-9709-d6b3deda2012\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fztgr" Oct 30 20:45:21.170477 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.170454 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4hh6\" (UniqueName: \"kubernetes.io/projected/bce11e0b-618d-40b0-9709-d6b3deda2012-kube-api-access-j4hh6\") pod \"payload-processing-55fbbbf9dc-fztgr\" (UID: \"bce11e0b-618d-40b0-9709-d6b3deda2012\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fztgr" Oct 30 20:45:21.170711 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.170694 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f69fh\" (UniqueName: \"kubernetes.io/projected/c7711661-bd4e-49ba-96f6-2739374b633e-kube-api-access-f69fh\") pod \"payload-pre-processing-7b98bb678c-shfhk\" (UID: \"c7711661-bd4e-49ba-96f6-2739374b633e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-shfhk" Oct 30 20:45:21.316253 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.316166 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-fztgr" Oct 30 20:45:21.342625 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.342595 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-shfhk" Oct 30 20:45:21.591614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.591586 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fztgr"] Oct 30 20:45:21.594872 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:45:21.594847 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce11e0b_618d_40b0_9709_d6b3deda2012.slice/crio-72c70eb408e720bc1952066d6d997357f1f0182a2d223aad4ce8c1043eae17f2 WatchSource:0}: Error finding container 72c70eb408e720bc1952066d6d997357f1f0182a2d223aad4ce8c1043eae17f2: Status 404 returned error can't find the container with id 72c70eb408e720bc1952066d6d997357f1f0182a2d223aad4ce8c1043eae17f2 Oct 30 20:45:21.631484 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.631406 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-shfhk"] Oct 30 20:45:21.634199 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:45:21.634167 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7711661_bd4e_49ba_96f6_2739374b633e.slice/crio-bfcca1ae27b0918eca565de149cffa24dcfbbc9eaddd2385a82757336223d130 WatchSource:0}: Error finding container bfcca1ae27b0918eca565de149cffa24dcfbbc9eaddd2385a82757336223d130: Status 404 returned error can't find the container with id bfcca1ae27b0918eca565de149cffa24dcfbbc9eaddd2385a82757336223d130 Oct 30 20:45:21.708304 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.708276 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" Oct 30 20:45:21.767616 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.767580 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/577190e1-ed57-4e93-9b81-e18111117ad0-kube-api-access-84cxm\" (UniqueName: \"kubernetes.io/projected/577190e1-ed57-4e93-9b81-e18111117ad0-kube-api-access-84cxm\") pod \"577190e1-ed57-4e93-9b81-e18111117ad0\" (UID: \"577190e1-ed57-4e93-9b81-e18111117ad0\") " Oct 30 20:45:21.767811 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.767662 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/577190e1-ed57-4e93-9b81-e18111117ad0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/577190e1-ed57-4e93-9b81-e18111117ad0-maas-api-service-ca\") pod \"577190e1-ed57-4e93-9b81-e18111117ad0\" (UID: \"577190e1-ed57-4e93-9b81-e18111117ad0\") " Oct 30 20:45:21.768056 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.768027 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577190e1-ed57-4e93-9b81-e18111117ad0-maas-api-service-ca" pod "577190e1-ed57-4e93-9b81-e18111117ad0" (UID: "577190e1-ed57-4e93-9b81-e18111117ad0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:45:21.769655 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.769627 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577190e1-ed57-4e93-9b81-e18111117ad0-kube-api-access-84cxm" pod "577190e1-ed57-4e93-9b81-e18111117ad0" (UID: "577190e1-ed57-4e93-9b81-e18111117ad0"). InnerVolumeSpecName "kube-api-access-84cxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:45:21.869071 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.868978 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84cxm\" (UniqueName: \"kubernetes.io/projected/577190e1-ed57-4e93-9b81-e18111117ad0-kube-api-access-84cxm\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:45:21.869071 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:21.869014 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/577190e1-ed57-4e93-9b81-e18111117ad0-maas-api-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:45:22.527109 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:45:22.527008 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:45:22.594931 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:22.594885 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-shfhk" event={"ID":"c7711661-bd4e-49ba-96f6-2739374b633e","Type":"ContainerStarted","Data":"bfcca1ae27b0918eca565de149cffa24dcfbbc9eaddd2385a82757336223d130"} Oct 30 20:45:22.596781 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:22.596745 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" Oct 30 20:45:22.596896 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:22.596817 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb" event={"ID":"577190e1-ed57-4e93-9b81-e18111117ad0","Type":"ContainerDied","Data":"79dd30773062a8189a7db6b2436c2c48778c5b62e1b2b86e8f997fd14e37b14d"} Oct 30 20:45:22.596896 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:22.596868 4848 scope.go:122] "RemoveContainer" containerID="aa13dd8e827e6308421f8aadd45b04c7bcda147e51e89f51e059dd10334f2194" Oct 30 20:45:22.602621 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:22.602594 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-fztgr" event={"ID":"bce11e0b-618d-40b0-9709-d6b3deda2012","Type":"ContainerStarted","Data":"72c70eb408e720bc1952066d6d997357f1f0182a2d223aad4ce8c1043eae17f2"} Oct 30 20:45:22.629446 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:22.629415 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb"] Oct 30 20:45:22.631779 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:22.631739 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889885-g6prb"] Oct 30 20:45:22.931601 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:22.931572 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="577190e1-ed57-4e93-9b81-e18111117ad0" path="/var/lib/kubelet/pods/577190e1-ed57-4e93-9b81-e18111117ad0/volumes" Oct 30 20:45:24.614871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:24.614834 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-shfhk" event={"ID":"c7711661-bd4e-49ba-96f6-2739374b633e","Type":"ContainerStarted","Data":"b87ee9a81b2c7e75f3a28f5770f69fe9ed29003346bcb8fe854c90744f18ccce"} Oct 30 20:45:24.615319 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:24.614903 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-shfhk" Oct 30 20:45:24.616338 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:24.616314 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-fztgr" event={"ID":"bce11e0b-618d-40b0-9709-d6b3deda2012","Type":"ContainerStarted","Data":"d1f43023783d18196c81d4d76ec11e50fcaaf14008d4f28c588744ce448e9653"} Oct 30 20:45:24.616457 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:24.616442 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-fztgr" Oct 30 20:45:24.634139 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:24.634094 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-shfhk" podStartSLOduration=1.327357149 podStartE2EDuration="3.634079821s" totalImagesPullingTime="2.306722672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:24.632359169 +0000 UTC m=+429.812255676" watchObservedRunningTime="2026-10-30 20:45:24.634079821 +0000 UTC m=+429.813976328" Oct 30 20:45:24.648786 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:24.648717 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-fztgr" podStartSLOduration=2.267881584 podStartE2EDuration="4.648700836s" totalImagesPullingTime="2.380819252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:24.648261629 +0000 UTC m=+429.828158136" watchObservedRunningTime="2026-10-30 20:45:24.648700836 +0000 UTC m=+429.828597343" Oct 30 20:45:30.584911 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:30.584880 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-m2n56" Oct 30 20:45:30.850268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:30.850195 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-b47bd7b99-gx47b"] Oct 30 20:45:30.850533 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:30.850521 4848 cpu_manager.go:414] "containerMap: removing container" podUID="577190e1-ed57-4e93-9b81-e18111117ad0" containerName="cleanup" Oct 30 20:45:30.850570 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:30.850534 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="577190e1-ed57-4e93-9b81-e18111117ad0" containerName="cleanup" Oct 30 20:45:30.850570 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:30.850562 4848 cpu_manager.go:414] "containerMap: removing container" podUID="577190e1-ed57-4e93-9b81-e18111117ad0" containerName="cleanup" Oct 30 20:45:30.850570 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:30.850567 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="577190e1-ed57-4e93-9b81-e18111117ad0" containerName="cleanup" Oct 30 20:45:30.850662 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:30.850573 4848 cpu_manager.go:414] "containerMap: removing container" podUID="577190e1-ed57-4e93-9b81-e18111117ad0" containerName="cleanup" Oct 30 20:45:30.850662 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:30.850579 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="577190e1-ed57-4e93-9b81-e18111117ad0" containerName="cleanup" Oct 30 20:45:30.850662 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:30.850634 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="577190e1-ed57-4e93-9b81-e18111117ad0" containerName="cleanup" Oct 30 20:45:30.850662 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:30.850640 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="577190e1-ed57-4e93-9b81-e18111117ad0" containerName="cleanup" Oct 30 20:45:30.850662 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:30.850649 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="577190e1-ed57-4e93-9b81-e18111117ad0" containerName="cleanup" Oct 30 20:45:30.853644 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:30.853625 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-b47bd7b99-gx47b" Oct 30 20:45:30.862280 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:30.862254 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-b47bd7b99-gx47b"] Oct 30 20:45:30.946210 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:30.946177 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01739bcb-1df4-4118-9b9f-2d52db7ed86b-metrics-certs\") pod \"ai-gateway-controller-b47bd7b99-gx47b\" (UID: \"01739bcb-1df4-4118-9b9f-2d52db7ed86b\") " pod="opendatahub/ai-gateway-controller-b47bd7b99-gx47b" Oct 30 20:45:30.946367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:30.946223 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdjzw\" (UniqueName: \"kubernetes.io/projected/01739bcb-1df4-4118-9b9f-2d52db7ed86b-kube-api-access-tdjzw\") pod \"ai-gateway-controller-b47bd7b99-gx47b\" (UID: \"01739bcb-1df4-4118-9b9f-2d52db7ed86b\") " pod="opendatahub/ai-gateway-controller-b47bd7b99-gx47b" Oct 30 20:45:31.046816 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:31.046786 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdjzw\" (UniqueName: \"kubernetes.io/projected/01739bcb-1df4-4118-9b9f-2d52db7ed86b-kube-api-access-tdjzw\") pod \"ai-gateway-controller-b47bd7b99-gx47b\" (UID: \"01739bcb-1df4-4118-9b9f-2d52db7ed86b\") " pod="opendatahub/ai-gateway-controller-b47bd7b99-gx47b" Oct 30 20:45:31.046904 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:31.046863 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01739bcb-1df4-4118-9b9f-2d52db7ed86b-metrics-certs\") pod \"ai-gateway-controller-b47bd7b99-gx47b\" (UID: \"01739bcb-1df4-4118-9b9f-2d52db7ed86b\") " pod="opendatahub/ai-gateway-controller-b47bd7b99-gx47b" Oct 30 20:45:31.049235 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:31.049208 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01739bcb-1df4-4118-9b9f-2d52db7ed86b-metrics-certs\") pod \"ai-gateway-controller-b47bd7b99-gx47b\" (UID: \"01739bcb-1df4-4118-9b9f-2d52db7ed86b\") " pod="opendatahub/ai-gateway-controller-b47bd7b99-gx47b" Oct 30 20:45:31.056049 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:31.056028 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdjzw\" (UniqueName: \"kubernetes.io/projected/01739bcb-1df4-4118-9b9f-2d52db7ed86b-kube-api-access-tdjzw\") pod \"ai-gateway-controller-b47bd7b99-gx47b\" (UID: \"01739bcb-1df4-4118-9b9f-2d52db7ed86b\") " pod="opendatahub/ai-gateway-controller-b47bd7b99-gx47b" Oct 30 20:45:31.166945 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:31.166880 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-b47bd7b99-gx47b" Oct 30 20:45:31.436574 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:31.436543 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-b47bd7b99-gx47b"] Oct 30 20:45:31.440380 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:45:31.440337 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01739bcb_1df4_4118_9b9f_2d52db7ed86b.slice/crio-147f756923d3ab0a9c09107c61b715662eed5ad04cc00dddae529a884315dd51 WatchSource:0}: Error finding container 147f756923d3ab0a9c09107c61b715662eed5ad04cc00dddae529a884315dd51: Status 404 returned error can't find the container with id 147f756923d3ab0a9c09107c61b715662eed5ad04cc00dddae529a884315dd51 Oct 30 20:45:31.648398 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:31.648351 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-b47bd7b99-gx47b" event={"ID":"01739bcb-1df4-4118-9b9f-2d52db7ed86b","Type":"ContainerStarted","Data":"147f756923d3ab0a9c09107c61b715662eed5ad04cc00dddae529a884315dd51"} Oct 30 20:45:32.653882 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:32.653841 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-b47bd7b99-gx47b" event={"ID":"01739bcb-1df4-4118-9b9f-2d52db7ed86b","Type":"ContainerStarted","Data":"2ef0dd7204cafe1ea8a5e4fa99225466e6bce15a815d23ceded754217597eebc"} Oct 30 20:45:32.654271 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:32.653957 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-b47bd7b99-gx47b" Oct 30 20:45:32.672580 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:32.672533 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-b47bd7b99-gx47b" podStartSLOduration=2.265349634 podStartE2EDuration="2.672521149s" totalImagesPullingTime="407.171515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:32.672154462 +0000 UTC m=+437.852050966" watchObservedRunningTime="2026-10-30 20:45:32.672521149 +0000 UTC m=+437.852417656" Oct 30 20:45:35.624384 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:35.624343 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-shfhk" Oct 30 20:45:35.624782 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:35.624429 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-fztgr" Oct 30 20:45:43.660897 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:43.660865 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-b47bd7b99-gx47b" Oct 30 20:45:43.738896 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:43.738866 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-m2n56"] Oct 30 20:45:43.739118 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:43.739096 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-m2n56" podUID="e57fcd7a-e48b-4935-96ea-0e8bf7d3258b" containerName="manager" containerID="cri-o://402578d1a5314529cca4c0ead61678bf120870d4b074ee132bb22b6a2f9bae36" gracePeriod=30 Oct 30 20:45:43.978953 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:43.978929 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-m2n56" Oct 30 20:45:44.057844 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:44.057809 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e57fcd7a-e48b-4935-96ea-0e8bf7d3258b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e57fcd7a-e48b-4935-96ea-0e8bf7d3258b-metrics-certs\") pod \"e57fcd7a-e48b-4935-96ea-0e8bf7d3258b\" (UID: \"e57fcd7a-e48b-4935-96ea-0e8bf7d3258b\") " Oct 30 20:45:44.058018 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:44.057891 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e57fcd7a-e48b-4935-96ea-0e8bf7d3258b-kube-api-access-ng468\" (UniqueName: \"kubernetes.io/projected/e57fcd7a-e48b-4935-96ea-0e8bf7d3258b-kube-api-access-ng468\") pod \"e57fcd7a-e48b-4935-96ea-0e8bf7d3258b\" (UID: \"e57fcd7a-e48b-4935-96ea-0e8bf7d3258b\") " Oct 30 20:45:44.059873 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:44.059844 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e57fcd7a-e48b-4935-96ea-0e8bf7d3258b-metrics-certs" pod "e57fcd7a-e48b-4935-96ea-0e8bf7d3258b" (UID: "e57fcd7a-e48b-4935-96ea-0e8bf7d3258b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:45:44.059973 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:44.059894 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57fcd7a-e48b-4935-96ea-0e8bf7d3258b-kube-api-access-ng468" pod "e57fcd7a-e48b-4935-96ea-0e8bf7d3258b" (UID: "e57fcd7a-e48b-4935-96ea-0e8bf7d3258b"). InnerVolumeSpecName "kube-api-access-ng468". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:45:44.159486 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:44.159460 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng468\" (UniqueName: \"kubernetes.io/projected/e57fcd7a-e48b-4935-96ea-0e8bf7d3258b-kube-api-access-ng468\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:45:44.159486 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:44.159485 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e57fcd7a-e48b-4935-96ea-0e8bf7d3258b-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:45:44.704721 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:44.704691 4848 generic.go:356] "Generic (PLEG): container finished" podID="e57fcd7a-e48b-4935-96ea-0e8bf7d3258b" containerID="402578d1a5314529cca4c0ead61678bf120870d4b074ee132bb22b6a2f9bae36" exitCode=0 Oct 30 20:45:44.705118 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:44.704758 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-m2n56" event={"ID":"e57fcd7a-e48b-4935-96ea-0e8bf7d3258b","Type":"ContainerDied","Data":"402578d1a5314529cca4c0ead61678bf120870d4b074ee132bb22b6a2f9bae36"} Oct 30 20:45:44.705118 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:44.704739 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-m2n56" Oct 30 20:45:44.705118 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:44.704813 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-m2n56" event={"ID":"e57fcd7a-e48b-4935-96ea-0e8bf7d3258b","Type":"ContainerDied","Data":"15f80510c6bb55c7e9bc4552e2a1dc8a57eca665b060677c1dfc3a23eba2b69c"} Oct 30 20:45:44.705118 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:44.704837 4848 scope.go:122] "RemoveContainer" containerID="402578d1a5314529cca4c0ead61678bf120870d4b074ee132bb22b6a2f9bae36" Oct 30 20:45:44.713606 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:44.713588 4848 scope.go:122] "RemoveContainer" containerID="402578d1a5314529cca4c0ead61678bf120870d4b074ee132bb22b6a2f9bae36" Oct 30 20:45:44.713872 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:45:44.713855 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"402578d1a5314529cca4c0ead61678bf120870d4b074ee132bb22b6a2f9bae36\": container with ID starting with 402578d1a5314529cca4c0ead61678bf120870d4b074ee132bb22b6a2f9bae36 not found: ID does not exist" containerID="402578d1a5314529cca4c0ead61678bf120870d4b074ee132bb22b6a2f9bae36" Oct 30 20:45:44.713932 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:44.713881 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"402578d1a5314529cca4c0ead61678bf120870d4b074ee132bb22b6a2f9bae36"} err="failed to get container status \"402578d1a5314529cca4c0ead61678bf120870d4b074ee132bb22b6a2f9bae36\": rpc error: code = NotFound desc = could not find container \"402578d1a5314529cca4c0ead61678bf120870d4b074ee132bb22b6a2f9bae36\": container with ID starting with 402578d1a5314529cca4c0ead61678bf120870d4b074ee132bb22b6a2f9bae36 not found: ID does not exist" Oct 30 20:45:44.726994 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:44.726971 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-m2n56"] Oct 30 20:45:44.731807 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:44.731788 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-m2n56"] Oct 30 20:45:44.929623 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:44.929592 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e57fcd7a-e48b-4935-96ea-0e8bf7d3258b" path="/var/lib/kubelet/pods/e57fcd7a-e48b-4935-96ea-0e8bf7d3258b/volumes" Oct 30 20:45:50.961675 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:50.961644 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87"] Oct 30 20:45:50.962198 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:50.962027 4848 cpu_manager.go:414] "containerMap: removing container" podUID="e57fcd7a-e48b-4935-96ea-0e8bf7d3258b" containerName="manager" Oct 30 20:45:50.962198 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:50.962039 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e57fcd7a-e48b-4935-96ea-0e8bf7d3258b" containerName="manager" Oct 30 20:45:50.962198 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:50.962090 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="e57fcd7a-e48b-4935-96ea-0e8bf7d3258b" containerName="manager" Oct 30 20:45:50.965085 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:50.965064 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:50.996982 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:50.996962 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 20:45:50.997344 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:50.997319 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 20:45:50.998335 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:50.998314 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-s86lq\"" Oct 30 20:45:50.998478 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:50.998391 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 20:45:51.000166 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.000142 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87"] Oct 30 20:45:51.119684 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.119652 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70e69039-4f0f-4f23-81cf-a84eb0bbf942-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87\" (UID: \"70e69039-4f0f-4f23-81cf-a84eb0bbf942\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:51.119684 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.119686 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70e69039-4f0f-4f23-81cf-a84eb0bbf942-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87\" (UID: \"70e69039-4f0f-4f23-81cf-a84eb0bbf942\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:51.119939 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.119705 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70e69039-4f0f-4f23-81cf-a84eb0bbf942-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87\" (UID: \"70e69039-4f0f-4f23-81cf-a84eb0bbf942\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:51.119939 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.119830 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70e69039-4f0f-4f23-81cf-a84eb0bbf942-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87\" (UID: \"70e69039-4f0f-4f23-81cf-a84eb0bbf942\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:51.119939 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.119890 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvvlk\" (UniqueName: \"kubernetes.io/projected/70e69039-4f0f-4f23-81cf-a84eb0bbf942-kube-api-access-mvvlk\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87\" (UID: \"70e69039-4f0f-4f23-81cf-a84eb0bbf942\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:51.120042 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.119958 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70e69039-4f0f-4f23-81cf-a84eb0bbf942-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87\" (UID: \"70e69039-4f0f-4f23-81cf-a84eb0bbf942\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:51.220462 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.220376 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70e69039-4f0f-4f23-81cf-a84eb0bbf942-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87\" (UID: \"70e69039-4f0f-4f23-81cf-a84eb0bbf942\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:51.220462 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.220422 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70e69039-4f0f-4f23-81cf-a84eb0bbf942-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87\" (UID: \"70e69039-4f0f-4f23-81cf-a84eb0bbf942\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:51.220462 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.220446 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70e69039-4f0f-4f23-81cf-a84eb0bbf942-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87\" (UID: \"70e69039-4f0f-4f23-81cf-a84eb0bbf942\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:51.220681 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.220492 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70e69039-4f0f-4f23-81cf-a84eb0bbf942-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87\" (UID: \"70e69039-4f0f-4f23-81cf-a84eb0bbf942\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:51.220681 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.220541 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvvlk\" (UniqueName: \"kubernetes.io/projected/70e69039-4f0f-4f23-81cf-a84eb0bbf942-kube-api-access-mvvlk\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87\" (UID: \"70e69039-4f0f-4f23-81cf-a84eb0bbf942\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:51.220681 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.220569 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70e69039-4f0f-4f23-81cf-a84eb0bbf942-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87\" (UID: \"70e69039-4f0f-4f23-81cf-a84eb0bbf942\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:51.220913 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.220886 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70e69039-4f0f-4f23-81cf-a84eb0bbf942-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87\" (UID: \"70e69039-4f0f-4f23-81cf-a84eb0bbf942\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:51.220981 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.220938 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70e69039-4f0f-4f23-81cf-a84eb0bbf942-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87\" (UID: \"70e69039-4f0f-4f23-81cf-a84eb0bbf942\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:51.221028 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.221001 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70e69039-4f0f-4f23-81cf-a84eb0bbf942-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87\" (UID: \"70e69039-4f0f-4f23-81cf-a84eb0bbf942\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:51.222793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.222733 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70e69039-4f0f-4f23-81cf-a84eb0bbf942-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87\" (UID: \"70e69039-4f0f-4f23-81cf-a84eb0bbf942\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:51.223000 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.222982 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70e69039-4f0f-4f23-81cf-a84eb0bbf942-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87\" (UID: \"70e69039-4f0f-4f23-81cf-a84eb0bbf942\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:51.229434 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.229411 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvvlk\" (UniqueName: \"kubernetes.io/projected/70e69039-4f0f-4f23-81cf-a84eb0bbf942-kube-api-access-mvvlk\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87\" (UID: \"70e69039-4f0f-4f23-81cf-a84eb0bbf942\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:51.277606 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.277563 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:51.565052 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.564972 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87"] Oct 30 20:45:51.568509 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:45:51.568480 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e69039_4f0f_4f23_81cf_a84eb0bbf942.slice/crio-188c8f8995fa25d64c3f607fd8640ac65df9e136e0a75f54601661081207dc40 WatchSource:0}: Error finding container 188c8f8995fa25d64c3f607fd8640ac65df9e136e0a75f54601661081207dc40: Status 404 returned error can't find the container with id 188c8f8995fa25d64c3f607fd8640ac65df9e136e0a75f54601661081207dc40 Oct 30 20:45:51.734423 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:51.734387 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" event={"ID":"70e69039-4f0f-4f23-81cf-a84eb0bbf942","Type":"ContainerStarted","Data":"188c8f8995fa25d64c3f607fd8640ac65df9e136e0a75f54601661081207dc40"} Oct 30 20:45:52.510038 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:45:52.510002 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:45:53.747501 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:53.747458 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" event={"ID":"70e69039-4f0f-4f23-81cf-a84eb0bbf942","Type":"ContainerStarted","Data":"75d812d938cfc9a51d740c53b273e14a81dd5f19f13482d516b8e6f4df4f7448"} Oct 30 20:45:53.768465 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:53.768417 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" podStartSLOduration=2.2827999 podStartE2EDuration="3.768403285s" totalImagesPullingTime="1.485603385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:53.76732343 +0000 UTC m=+458.947219936" watchObservedRunningTime="2026-10-30 20:45:53.768403285 +0000 UTC m=+458.948299793" Oct 30 20:45:54.278377 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:54.278344 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:45:55.241117 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.241087 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj"] Oct 30 20:45:55.244484 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.244467 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:45:55.252874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.252851 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 20:45:55.264234 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.264203 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d640ea-9cc3-4090-a65e-972250c29b28-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vgkmj\" (UID: \"d4d640ea-9cc3-4090-a65e-972250c29b28\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:45:55.264416 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.264249 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d640ea-9cc3-4090-a65e-972250c29b28-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vgkmj\" (UID: \"d4d640ea-9cc3-4090-a65e-972250c29b28\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:45:55.264416 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.264301 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d640ea-9cc3-4090-a65e-972250c29b28-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vgkmj\" (UID: \"d4d640ea-9cc3-4090-a65e-972250c29b28\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:45:55.264525 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.264431 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7bjs\" (UniqueName: \"kubernetes.io/projected/d4d640ea-9cc3-4090-a65e-972250c29b28-kube-api-access-f7bjs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vgkmj\" (UID: \"d4d640ea-9cc3-4090-a65e-972250c29b28\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:45:55.264586 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.264550 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d640ea-9cc3-4090-a65e-972250c29b28-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vgkmj\" (UID: \"d4d640ea-9cc3-4090-a65e-972250c29b28\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:45:55.264639 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.264588 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d640ea-9cc3-4090-a65e-972250c29b28-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vgkmj\" (UID: \"d4d640ea-9cc3-4090-a65e-972250c29b28\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:45:55.265486 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.265460 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj"] Oct 30 20:45:55.365816 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.365762 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7bjs\" (UniqueName: \"kubernetes.io/projected/d4d640ea-9cc3-4090-a65e-972250c29b28-kube-api-access-f7bjs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vgkmj\" (UID: \"d4d640ea-9cc3-4090-a65e-972250c29b28\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:45:55.365980 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.365870 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d640ea-9cc3-4090-a65e-972250c29b28-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vgkmj\" (UID: \"d4d640ea-9cc3-4090-a65e-972250c29b28\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:45:55.365980 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.365892 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d640ea-9cc3-4090-a65e-972250c29b28-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vgkmj\" (UID: \"d4d640ea-9cc3-4090-a65e-972250c29b28\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:45:55.366064 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.366013 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d640ea-9cc3-4090-a65e-972250c29b28-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vgkmj\" (UID: \"d4d640ea-9cc3-4090-a65e-972250c29b28\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:45:55.366064 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.366046 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d640ea-9cc3-4090-a65e-972250c29b28-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vgkmj\" (UID: \"d4d640ea-9cc3-4090-a65e-972250c29b28\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:45:55.366132 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.366072 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d640ea-9cc3-4090-a65e-972250c29b28-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vgkmj\" (UID: \"d4d640ea-9cc3-4090-a65e-972250c29b28\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:45:55.366307 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.366288 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4d640ea-9cc3-4090-a65e-972250c29b28-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vgkmj\" (UID: \"d4d640ea-9cc3-4090-a65e-972250c29b28\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:45:55.366411 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.366393 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4d640ea-9cc3-4090-a65e-972250c29b28-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vgkmj\" (UID: \"d4d640ea-9cc3-4090-a65e-972250c29b28\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:45:55.366456 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.366431 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4d640ea-9cc3-4090-a65e-972250c29b28-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vgkmj\" (UID: \"d4d640ea-9cc3-4090-a65e-972250c29b28\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:45:55.368111 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.368095 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4d640ea-9cc3-4090-a65e-972250c29b28-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vgkmj\" (UID: \"d4d640ea-9cc3-4090-a65e-972250c29b28\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:45:55.368283 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.368268 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4d640ea-9cc3-4090-a65e-972250c29b28-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vgkmj\" (UID: \"d4d640ea-9cc3-4090-a65e-972250c29b28\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:45:55.376002 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.375983 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7bjs\" (UniqueName: \"kubernetes.io/projected/d4d640ea-9cc3-4090-a65e-972250c29b28-kube-api-access-f7bjs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-vgkmj\" (UID: \"d4d640ea-9cc3-4090-a65e-972250c29b28\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:45:55.557951 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.557875 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:45:55.840544 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:55.840523 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj"] Oct 30 20:45:55.841992 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:45:55.841965 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d640ea_9cc3_4090_a65e_972250c29b28.slice/crio-adca9875ea02a7548d2069e1e85a723e842bd96c0356a746295d5a2c184a4858 WatchSource:0}: Error finding container adca9875ea02a7548d2069e1e85a723e842bd96c0356a746295d5a2c184a4858: Status 404 returned error can't find the container with id adca9875ea02a7548d2069e1e85a723e842bd96c0356a746295d5a2c184a4858 Oct 30 20:45:56.762560 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:56.762511 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" event={"ID":"d4d640ea-9cc3-4090-a65e-972250c29b28","Type":"ContainerStarted","Data":"632790f8bcc0b456651709e6704f494dbedfcd18ac563d09c95e9e9a36ae647f"} Oct 30 20:45:56.763033 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:56.762601 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" event={"ID":"d4d640ea-9cc3-4090-a65e-972250c29b28","Type":"ContainerStarted","Data":"adca9875ea02a7548d2069e1e85a723e842bd96c0356a746295d5a2c184a4858"} Oct 30 20:45:56.794159 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:56.794096 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" podStartSLOduration=1.466860453 podStartE2EDuration="1.794059667s" totalImagesPullingTime="327.199214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:45:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:45:56.793432546 +0000 UTC m=+461.973329053" watchObservedRunningTime="2026-10-30 20:45:56.794059667 +0000 UTC m=+461.973956174" Oct 30 20:45:57.558497 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:45:57.558463 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:46:01.277934 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:01.277900 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:46:01.290851 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:01.290819 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:46:01.793737 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:01.793704 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87" Oct 30 20:46:02.025985 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.025952 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-gtxbn"] Oct 30 20:46:02.034295 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.034270 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:02.039989 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.039967 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 20:46:02.069994 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.069916 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-gtxbn"] Oct 30 20:46:02.129338 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.129298 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ef1876-cf8b-4383-9757-6569689a6978-home\") pod \"echo-simulator-kserve-64b6f8f96c-gtxbn\" (UID: \"15ef1876-cf8b-4383-9757-6569689a6978\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:02.129532 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.129385 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ef1876-cf8b-4383-9757-6569689a6978-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-gtxbn\" (UID: \"15ef1876-cf8b-4383-9757-6569689a6978\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:02.129532 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.129485 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz2bh\" (UniqueName: \"kubernetes.io/projected/15ef1876-cf8b-4383-9757-6569689a6978-kube-api-access-kz2bh\") pod \"echo-simulator-kserve-64b6f8f96c-gtxbn\" (UID: \"15ef1876-cf8b-4383-9757-6569689a6978\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:02.129532 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.129507 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ef1876-cf8b-4383-9757-6569689a6978-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-gtxbn\" (UID: \"15ef1876-cf8b-4383-9757-6569689a6978\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:02.129709 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.129547 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ef1876-cf8b-4383-9757-6569689a6978-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-gtxbn\" (UID: \"15ef1876-cf8b-4383-9757-6569689a6978\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:02.129709 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.129635 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ef1876-cf8b-4383-9757-6569689a6978-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-gtxbn\" (UID: \"15ef1876-cf8b-4383-9757-6569689a6978\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:02.230812 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.230759 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ef1876-cf8b-4383-9757-6569689a6978-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-gtxbn\" (UID: \"15ef1876-cf8b-4383-9757-6569689a6978\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:02.230991 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.230845 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz2bh\" (UniqueName: \"kubernetes.io/projected/15ef1876-cf8b-4383-9757-6569689a6978-kube-api-access-kz2bh\") pod \"echo-simulator-kserve-64b6f8f96c-gtxbn\" (UID: \"15ef1876-cf8b-4383-9757-6569689a6978\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:02.230991 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.230877 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ef1876-cf8b-4383-9757-6569689a6978-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-gtxbn\" (UID: \"15ef1876-cf8b-4383-9757-6569689a6978\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:02.230991 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.230929 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ef1876-cf8b-4383-9757-6569689a6978-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-gtxbn\" (UID: \"15ef1876-cf8b-4383-9757-6569689a6978\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:02.230991 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.230973 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ef1876-cf8b-4383-9757-6569689a6978-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-gtxbn\" (UID: \"15ef1876-cf8b-4383-9757-6569689a6978\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:02.231195 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.231115 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ef1876-cf8b-4383-9757-6569689a6978-home\") pod \"echo-simulator-kserve-64b6f8f96c-gtxbn\" (UID: \"15ef1876-cf8b-4383-9757-6569689a6978\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:02.231384 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.231358 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15ef1876-cf8b-4383-9757-6569689a6978-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-gtxbn\" (UID: \"15ef1876-cf8b-4383-9757-6569689a6978\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:02.231384 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.231375 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15ef1876-cf8b-4383-9757-6569689a6978-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-gtxbn\" (UID: \"15ef1876-cf8b-4383-9757-6569689a6978\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:02.231498 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.231466 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15ef1876-cf8b-4383-9757-6569689a6978-home\") pod \"echo-simulator-kserve-64b6f8f96c-gtxbn\" (UID: \"15ef1876-cf8b-4383-9757-6569689a6978\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:02.233204 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.233173 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15ef1876-cf8b-4383-9757-6569689a6978-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-gtxbn\" (UID: \"15ef1876-cf8b-4383-9757-6569689a6978\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:02.233441 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.233424 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15ef1876-cf8b-4383-9757-6569689a6978-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-gtxbn\" (UID: \"15ef1876-cf8b-4383-9757-6569689a6978\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:02.244317 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.244291 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz2bh\" (UniqueName: \"kubernetes.io/projected/15ef1876-cf8b-4383-9757-6569689a6978-kube-api-access-kz2bh\") pod \"echo-simulator-kserve-64b6f8f96c-gtxbn\" (UID: \"15ef1876-cf8b-4383-9757-6569689a6978\") " pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:02.348176 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.348086 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:02.625651 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.625619 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-gtxbn"] Oct 30 20:46:02.628257 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:46:02.628233 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ef1876_cf8b_4383_9757_6569689a6978.slice/crio-340821b0d5b80c28366ba4e542d89cb770c444047271ddbb54e6ec5ab6ef8193 WatchSource:0}: Error finding container 340821b0d5b80c28366ba4e542d89cb770c444047271ddbb54e6ec5ab6ef8193: Status 404 returned error can't find the container with id 340821b0d5b80c28366ba4e542d89cb770c444047271ddbb54e6ec5ab6ef8193 Oct 30 20:46:02.788693 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:02.788657 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" event={"ID":"15ef1876-cf8b-4383-9757-6569689a6978","Type":"ContainerStarted","Data":"340821b0d5b80c28366ba4e542d89cb770c444047271ddbb54e6ec5ab6ef8193"} Oct 30 20:46:03.455306 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.455270 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f"] Oct 30 20:46:03.459678 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.459255 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:03.466901 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.466551 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 20:46:03.472411 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.472210 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f"] Oct 30 20:46:03.543789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.543593 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d257f\" (UID: \"ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:03.543789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.543695 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d257f\" (UID: \"ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:03.544037 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.543806 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d257f\" (UID: \"ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:03.544037 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.543836 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d257f\" (UID: \"ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:03.544037 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.543865 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d257f\" (UID: \"ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:03.544037 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.543900 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv5vw\" (UniqueName: \"kubernetes.io/projected/ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76-kube-api-access-lv5vw\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d257f\" (UID: \"ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:03.645391 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.645337 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d257f\" (UID: \"ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:03.645391 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.645395 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d257f\" (UID: \"ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:03.645604 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.645462 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d257f\" (UID: \"ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:03.645604 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.645486 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d257f\" (UID: \"ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:03.645604 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.645512 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d257f\" (UID: \"ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:03.645604 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.645542 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv5vw\" (UniqueName: \"kubernetes.io/projected/ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76-kube-api-access-lv5vw\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d257f\" (UID: \"ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:03.646141 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.646123 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d257f\" (UID: \"ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:03.646593 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.646556 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d257f\" (UID: \"ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:03.646816 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.646792 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d257f\" (UID: \"ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:03.648616 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.648573 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d257f\" (UID: \"ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:03.649367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.649347 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d257f\" (UID: \"ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:03.667576 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.667546 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv5vw\" (UniqueName: \"kubernetes.io/projected/ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76-kube-api-access-lv5vw\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-d257f\" (UID: \"ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:03.779994 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:03.779914 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:04.101931 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:04.101842 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f"] Oct 30 20:46:04.107006 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:46:04.106966 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7f6e4c_0b8f_4a83_bce8_5a0e4d937f76.slice/crio-bb9c0de2eaacb06e6a6fb6172d1d5d01348b279cd1373f0cecfa89e8f7bf3e2f WatchSource:0}: Error finding container bb9c0de2eaacb06e6a6fb6172d1d5d01348b279cd1373f0cecfa89e8f7bf3e2f: Status 404 returned error can't find the container with id bb9c0de2eaacb06e6a6fb6172d1d5d01348b279cd1373f0cecfa89e8f7bf3e2f Oct 30 20:46:04.807406 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:04.807370 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" event={"ID":"ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76","Type":"ContainerStarted","Data":"a6c6d2fac14c70c45337fe59651f1d9ea0db2f853ddfa92c0d05f706977e6921"} Oct 30 20:46:04.807857 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:04.807438 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" event={"ID":"ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76","Type":"ContainerStarted","Data":"bb9c0de2eaacb06e6a6fb6172d1d5d01348b279cd1373f0cecfa89e8f7bf3e2f"} Oct 30 20:46:04.809044 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:04.809010 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" event={"ID":"15ef1876-cf8b-4383-9757-6569689a6978","Type":"ContainerStarted","Data":"734e77b43f37bbae7877e9475994324198440c0a37d1172cd60f1541e9936bca"} Oct 30 20:46:04.826848 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:04.826805 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" podStartSLOduration=1.501890036 podStartE2EDuration="1.826792942s" totalImagesPullingTime="324.902906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:04.825346871 +0000 UTC m=+470.005243378" watchObservedRunningTime="2026-10-30 20:46:04.826792942 +0000 UTC m=+470.006689446" Oct 30 20:46:04.844618 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:04.844565 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" podStartSLOduration=1.822116646 podStartE2EDuration="3.844548558s" totalImagesPullingTime="2.022431912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:04.844271163 +0000 UTC m=+470.024167670" watchObservedRunningTime="2026-10-30 20:46:04.844548558 +0000 UTC m=+470.024445068" Oct 30 20:46:05.348393 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:05.348359 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:05.558129 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:05.558089 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:46:05.574504 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:05.574469 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:46:05.781137 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:05.781088 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:05.825665 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:05.825629 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-vgkmj" Oct 30 20:46:06.941579 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:06.941546 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj"] Oct 30 20:46:06.945209 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:06.945194 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:06.956502 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:06.956476 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h8j6c\"" Oct 30 20:46:06.968932 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:06.968904 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj"] Oct 30 20:46:06.976412 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:06.976373 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:06.976555 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:06.976425 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:06.976555 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:06.976527 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:06.976683 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:06.976592 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:06.976683 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:06.976636 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:06.976824 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:06.976709 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:06.976824 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:06.976735 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:06.976824 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:06.976754 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:06.976989 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:06.976905 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j77z7\" (UniqueName: \"kubernetes.io/projected/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-kube-api-access-j77z7\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:07.078317 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.078281 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:07.078317 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.078317 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:07.078530 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.078334 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:07.078530 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.078371 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j77z7\" (UniqueName: \"kubernetes.io/projected/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-kube-api-access-j77z7\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:07.078530 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.078410 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:07.078530 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.078439 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:07.078530 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.078488 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:07.078530 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.078527 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:07.078865 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.078801 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:07.078865 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.078836 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:07.078960 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.078893 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:07.078960 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.078928 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:07.079074 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.078975 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:07.079239 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.079216 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:07.080881 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.080860 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:07.080976 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.080960 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:07.087566 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.087548 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:07.087881 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.087855 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j77z7\" (UniqueName: \"kubernetes.io/projected/7dc1d3fc-a66d-41d9-b81c-d17801d8702e-kube-api-access-j77z7\") pod \"maas-default-gateway-openshift-default-77f556d647-fmzmj\" (UID: \"7dc1d3fc-a66d-41d9-b81c-d17801d8702e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:07.258509 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.258413 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:07.536216 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.536162 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj"] Oct 30 20:46:07.538496 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:46:07.538464 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc1d3fc_a66d_41d9_b81c_d17801d8702e.slice/crio-67cdda68114bddc7aff36cc303eab1c1bfe97266f234054df5b432f8895c1edf WatchSource:0}: Error finding container 67cdda68114bddc7aff36cc303eab1c1bfe97266f234054df5b432f8895c1edf: Status 404 returned error can't find the container with id 67cdda68114bddc7aff36cc303eab1c1bfe97266f234054df5b432f8895c1edf Oct 30 20:46:07.824187 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:07.824105 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" event={"ID":"7dc1d3fc-a66d-41d9-b81c-d17801d8702e","Type":"ContainerStarted","Data":"67cdda68114bddc7aff36cc303eab1c1bfe97266f234054df5b432f8895c1edf"} Oct 30 20:46:10.735301 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:10.735269 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:46:10.735551 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:10.735339 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:46:10.735551 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:10.735367 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:46:10.849414 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:10.849365 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" event={"ID":"7dc1d3fc-a66d-41d9-b81c-d17801d8702e","Type":"ContainerStarted","Data":"1fdac2088e1a3fa517e7ce9377a593c15ef908c1c9073d07b6c344f26c6944f6"} Oct 30 20:46:10.901423 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:10.901365 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" podStartSLOduration=1.707415382 podStartE2EDuration="4.901350168s" totalImagesPullingTime="3.193934786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:10.899009567 +0000 UTC m=+476.078906073" watchObservedRunningTime="2026-10-30 20:46:10.901350168 +0000 UTC m=+476.081246674" Oct 30 20:46:11.258858 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:11.258819 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:11.260300 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:11.260275 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fmzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 20:46:11.260422 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:11.260336 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" podUID="7dc1d3fc-a66d-41d9-b81c-d17801d8702e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 20:46:12.259202 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:12.259161 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fmzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 20:46:12.259586 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:12.259220 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" podUID="7dc1d3fc-a66d-41d9-b81c-d17801d8702e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 20:46:12.349210 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:12.349164 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:12.361988 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:12.361960 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:12.870578 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:12.870549 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-gtxbn" Oct 30 20:46:13.258942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:13.258903 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fmzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 20:46:13.259093 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:13.258961 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" podUID="7dc1d3fc-a66d-41d9-b81c-d17801d8702e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 20:46:13.780455 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:13.780419 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:13.792960 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:13.792937 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:13.874507 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:13.874480 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-d257f" Oct 30 20:46:14.259578 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:14.259539 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fmzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 20:46:14.259728 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:14.259600 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" podUID="7dc1d3fc-a66d-41d9-b81c-d17801d8702e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 20:46:15.259546 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:15.259462 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fmzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 20:46:15.259546 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:15.259528 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" podUID="7dc1d3fc-a66d-41d9-b81c-d17801d8702e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 20:46:16.259544 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:16.259508 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fmzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 20:46:16.259984 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:16.259562 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" podUID="7dc1d3fc-a66d-41d9-b81c-d17801d8702e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 20:46:17.259018 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:17.258980 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fmzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 20:46:17.259166 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:17.259030 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" podUID="7dc1d3fc-a66d-41d9-b81c-d17801d8702e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 20:46:18.224834 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:18.224792 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f767f5f8-dnjx6"] Oct 30 20:46:18.228416 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:18.228394 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f767f5f8-dnjx6" Oct 30 20:46:18.237536 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:18.237511 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f767f5f8-dnjx6"] Oct 30 20:46:18.259458 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:18.259420 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fmzmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 20:46:18.259608 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:18.259489 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" podUID="7dc1d3fc-a66d-41d9-b81c-d17801d8702e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 20:46:18.288649 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:18.288618 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dff97c85-a942-4a38-a233-a6563aa78d4d-tls-cert\") pod \"authorino-6f767f5f8-dnjx6\" (UID: \"dff97c85-a942-4a38-a233-a6563aa78d4d\") " pod="kuadrant-system/authorino-6f767f5f8-dnjx6" Oct 30 20:46:18.288760 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:18.288658 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w872\" (UniqueName: \"kubernetes.io/projected/dff97c85-a942-4a38-a233-a6563aa78d4d-kube-api-access-7w872\") pod \"authorino-6f767f5f8-dnjx6\" (UID: \"dff97c85-a942-4a38-a233-a6563aa78d4d\") " pod="kuadrant-system/authorino-6f767f5f8-dnjx6" Oct 30 20:46:18.389237 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:18.389190 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dff97c85-a942-4a38-a233-a6563aa78d4d-tls-cert\") pod \"authorino-6f767f5f8-dnjx6\" (UID: \"dff97c85-a942-4a38-a233-a6563aa78d4d\") " pod="kuadrant-system/authorino-6f767f5f8-dnjx6" Oct 30 20:46:18.389237 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:18.389230 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w872\" (UniqueName: \"kubernetes.io/projected/dff97c85-a942-4a38-a233-a6563aa78d4d-kube-api-access-7w872\") pod \"authorino-6f767f5f8-dnjx6\" (UID: \"dff97c85-a942-4a38-a233-a6563aa78d4d\") " pod="kuadrant-system/authorino-6f767f5f8-dnjx6" Oct 30 20:46:18.391708 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:18.391680 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dff97c85-a942-4a38-a233-a6563aa78d4d-tls-cert\") pod \"authorino-6f767f5f8-dnjx6\" (UID: \"dff97c85-a942-4a38-a233-a6563aa78d4d\") " pod="kuadrant-system/authorino-6f767f5f8-dnjx6" Oct 30 20:46:18.398852 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:18.398824 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w872\" (UniqueName: \"kubernetes.io/projected/dff97c85-a942-4a38-a233-a6563aa78d4d-kube-api-access-7w872\") pod \"authorino-6f767f5f8-dnjx6\" (UID: \"dff97c85-a942-4a38-a233-a6563aa78d4d\") " pod="kuadrant-system/authorino-6f767f5f8-dnjx6" Oct 30 20:46:18.540876 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:18.540723 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f767f5f8-dnjx6" Oct 30 20:46:18.848950 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:18.848921 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f767f5f8-dnjx6"] Oct 30 20:46:18.851028 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:46:18.851002 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddff97c85_a942_4a38_a233_a6563aa78d4d.slice/crio-184624684cfccc3e8d8a9cfedc3ad65ab84d3addd66a627db00afb12f9700712 WatchSource:0}: Error finding container 184624684cfccc3e8d8a9cfedc3ad65ab84d3addd66a627db00afb12f9700712: Status 404 returned error can't find the container with id 184624684cfccc3e8d8a9cfedc3ad65ab84d3addd66a627db00afb12f9700712 Oct 30 20:46:18.889619 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:18.889573 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f767f5f8-dnjx6" event={"ID":"dff97c85-a942-4a38-a233-a6563aa78d4d","Type":"ContainerStarted","Data":"184624684cfccc3e8d8a9cfedc3ad65ab84d3addd66a627db00afb12f9700712"} Oct 30 20:46:19.263564 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:19.263535 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:19.263980 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:19.263831 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:19.264492 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:19.264472 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" Oct 30 20:46:19.897414 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:19.897363 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f767f5f8-dnjx6" event={"ID":"dff97c85-a942-4a38-a233-a6563aa78d4d","Type":"ContainerStarted","Data":"b5b325f5f77e4c5bc0be1119742ef9c155c65b2d8d2dfde2f788de645ef7ad06"} Oct 30 20:46:19.917267 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:19.917200 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f767f5f8-dnjx6" podStartSLOduration=1.022204807 podStartE2EDuration="1.917187587s" totalImagesPullingTime="894.98278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:19.914169175 +0000 UTC m=+485.094065681" watchObservedRunningTime="2026-10-30 20:46:19.917187587 +0000 UTC m=+485.097084094" Oct 30 20:46:19.963275 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:19.963239 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-798dd5cc85-s2vpq"] Oct 30 20:46:19.963495 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:19.963465 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-798dd5cc85-s2vpq" podUID="54b58047-e429-408e-9376-9697f523b381" containerName="authorino" containerID="cri-o://ad66066ab381e0e15549b2d83539adf4642e5f400d40e6f6c18e1bece4600675" gracePeriod=30 Oct 30 20:46:20.222478 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:20.222452 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-798dd5cc85-s2vpq" Oct 30 20:46:20.307313 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:20.307229 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54b58047-e429-408e-9376-9697f523b381-tls-cert\" (UniqueName: \"kubernetes.io/secret/54b58047-e429-408e-9376-9697f523b381-tls-cert\") pod \"54b58047-e429-408e-9376-9697f523b381\" (UID: \"54b58047-e429-408e-9376-9697f523b381\") " Oct 30 20:46:20.307313 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:20.307295 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b58047-e429-408e-9376-9697f523b381-kube-api-access-jph6m\" (UniqueName: \"kubernetes.io/projected/54b58047-e429-408e-9376-9697f523b381-kube-api-access-jph6m\") pod \"54b58047-e429-408e-9376-9697f523b381\" (UID: \"54b58047-e429-408e-9376-9697f523b381\") " Oct 30 20:46:20.309681 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:20.309648 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b58047-e429-408e-9376-9697f523b381-kube-api-access-jph6m" pod "54b58047-e429-408e-9376-9697f523b381" (UID: "54b58047-e429-408e-9376-9697f523b381"). InnerVolumeSpecName "kube-api-access-jph6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:46:20.318607 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:20.318577 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54b58047-e429-408e-9376-9697f523b381-tls-cert" pod "54b58047-e429-408e-9376-9697f523b381" (UID: "54b58047-e429-408e-9376-9697f523b381"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:46:20.408448 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:20.408401 4848 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/54b58047-e429-408e-9376-9697f523b381-tls-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:46:20.408448 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:20.408443 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jph6m\" (UniqueName: \"kubernetes.io/projected/54b58047-e429-408e-9376-9697f523b381-kube-api-access-jph6m\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:46:20.904667 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:20.904632 4848 generic.go:356] "Generic (PLEG): container finished" podID="54b58047-e429-408e-9376-9697f523b381" containerID="ad66066ab381e0e15549b2d83539adf4642e5f400d40e6f6c18e1bece4600675" exitCode=0 Oct 30 20:46:20.905340 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:20.905319 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-798dd5cc85-s2vpq" Oct 30 20:46:20.912783 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:20.912746 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-798dd5cc85-s2vpq" event={"ID":"54b58047-e429-408e-9376-9697f523b381","Type":"ContainerDied","Data":"ad66066ab381e0e15549b2d83539adf4642e5f400d40e6f6c18e1bece4600675"} Oct 30 20:46:20.912947 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:20.912929 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-798dd5cc85-s2vpq" event={"ID":"54b58047-e429-408e-9376-9697f523b381","Type":"ContainerDied","Data":"938bd8bb82aa83d62f218a9fe773cf1b0af29cbacb1abe51f30fec2c29085332"} Oct 30 20:46:20.913070 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:20.913057 4848 scope.go:122] "RemoveContainer" containerID="ad66066ab381e0e15549b2d83539adf4642e5f400d40e6f6c18e1bece4600675" Oct 30 20:46:20.936347 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:20.936266 4848 scope.go:122] "RemoveContainer" containerID="ad66066ab381e0e15549b2d83539adf4642e5f400d40e6f6c18e1bece4600675" Oct 30 20:46:20.937346 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:46:20.937210 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad66066ab381e0e15549b2d83539adf4642e5f400d40e6f6c18e1bece4600675\": container with ID starting with ad66066ab381e0e15549b2d83539adf4642e5f400d40e6f6c18e1bece4600675 not found: ID does not exist" containerID="ad66066ab381e0e15549b2d83539adf4642e5f400d40e6f6c18e1bece4600675" Oct 30 20:46:20.937346 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:20.937249 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad66066ab381e0e15549b2d83539adf4642e5f400d40e6f6c18e1bece4600675"} err="failed to get container status \"ad66066ab381e0e15549b2d83539adf4642e5f400d40e6f6c18e1bece4600675\": rpc error: code = NotFound desc = could not find container \"ad66066ab381e0e15549b2d83539adf4642e5f400d40e6f6c18e1bece4600675\": container with ID starting with ad66066ab381e0e15549b2d83539adf4642e5f400d40e6f6c18e1bece4600675 not found: ID does not exist" Oct 30 20:46:20.954691 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:20.954659 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-798dd5cc85-s2vpq"] Oct 30 20:46:20.970074 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:20.970036 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-798dd5cc85-s2vpq"] Oct 30 20:46:22.511527 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:46:22.511490 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:46:22.930729 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:22.930695 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54b58047-e429-408e-9376-9697f523b381" path="/var/lib/kubelet/pods/54b58047-e429-408e-9376-9697f523b381/volumes" Oct 30 20:46:52.513454 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:46:52.513413 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:46:53.175013 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.174980 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq"] Oct 30 20:46:53.175378 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.175367 4848 cpu_manager.go:414] "containerMap: removing container" podUID="54b58047-e429-408e-9376-9697f523b381" containerName="authorino" Oct 30 20:46:53.175422 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.175380 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b58047-e429-408e-9376-9697f523b381" containerName="authorino" Oct 30 20:46:53.175456 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.175447 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b58047-e429-408e-9376-9697f523b381" containerName="authorino" Oct 30 20:46:53.178800 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.178782 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.185649 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.185271 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b612c7f2-openshift-default-dockercfg-jfvx8\"" Oct 30 20:46:53.214541 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.214515 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dshzp\" (UniqueName: \"kubernetes.io/projected/fd9df289-7dc4-4c00-9b75-d57dd17656f4-kube-api-access-dshzp\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.214672 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.214549 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-credential-socket\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.214672 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.214583 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istiod-ca-cert\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.214749 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.214667 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-token\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.214749 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.214710 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-workload-socket\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.214749 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.214725 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-data\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.214874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.214819 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-workload-certs\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.214874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.214839 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-podinfo\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.214874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.214866 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-envoy\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.257751 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.257716 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq"] Oct 30 20:46:53.315531 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.315498 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-workload-certs\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.315531 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.315532 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-podinfo\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.315801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.315553 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-envoy\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.315801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.315593 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dshzp\" (UniqueName: \"kubernetes.io/projected/fd9df289-7dc4-4c00-9b75-d57dd17656f4-kube-api-access-dshzp\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.315801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.315615 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-credential-socket\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.315801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.315670 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istiod-ca-cert\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.315801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.315709 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-token\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.315801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.315760 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-workload-socket\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.315801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.315796 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-data\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.316290 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.316260 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-credential-socket\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.316428 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.316384 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-workload-certs\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.316528 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.316505 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istiod-ca-cert\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.316528 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.316513 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-data\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.316790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.316751 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-workload-socket\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.318024 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.318004 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-envoy\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.318166 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.318147 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-podinfo\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.326338 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.326277 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dshzp\" (UniqueName: \"kubernetes.io/projected/fd9df289-7dc4-4c00-9b75-d57dd17656f4-kube-api-access-dshzp\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.327793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.327759 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-token\") pod \"e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.467351 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.467275 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9"] Oct 30 20:46:53.471939 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.471920 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.476376 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.476354 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-8f46c771-gw-openshift-default-dockercfg-g78lq\"" Oct 30 20:46:53.491256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.491239 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:53.494634 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.494617 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9"] Oct 30 20:46:53.518177 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.518147 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgsdb\" (UniqueName: \"kubernetes.io/projected/2a2a74d1-f1a1-4434-bc2f-966fb5123180-kube-api-access-cgsdb\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.518529 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.518271 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-workload-socket\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.518529 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.518308 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-token\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.518529 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.518334 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-data\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.518529 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.518430 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-envoy\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.518529 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.518453 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istiod-ca-cert\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.518529 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.518479 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-workload-certs\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.518747 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.518533 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-credential-socket\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.518747 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.518576 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-podinfo\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.619484 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.619451 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgsdb\" (UniqueName: \"kubernetes.io/projected/2a2a74d1-f1a1-4434-bc2f-966fb5123180-kube-api-access-cgsdb\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.619808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.619506 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-workload-socket\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.619808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.619532 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-token\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.619808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.619562 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-data\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.619808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.619588 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-envoy\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.619808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.619606 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istiod-ca-cert\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.619808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.619635 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-workload-certs\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.619808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.619684 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-credential-socket\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.619808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.619709 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-podinfo\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.620165 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.619970 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-workload-socket\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.620165 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.620016 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-data\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.620240 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.620207 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-workload-certs\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.620240 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.620228 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-credential-socket\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.620350 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.620332 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istiod-ca-cert\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.621938 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.621919 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-envoy\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.622290 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.622275 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-podinfo\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.629616 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.629596 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-token\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.633144 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.633128 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgsdb\" (UniqueName: \"kubernetes.io/projected/2a2a74d1-f1a1-4434-bc2f-966fb5123180-kube-api-access-cgsdb\") pod \"e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:53.778804 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.778756 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq"] Oct 30 20:46:53.780709 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:46:53.780683 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9df289_7dc4_4c00_9b75_d57dd17656f4.slice/crio-dc06c2aadd6142885d68c29fb0aba6b46bc011827d3bb59968cfdda5a46c4a2e WatchSource:0}: Error finding container dc06c2aadd6142885d68c29fb0aba6b46bc011827d3bb59968cfdda5a46c4a2e: Status 404 returned error can't find the container with id dc06c2aadd6142885d68c29fb0aba6b46bc011827d3bb59968cfdda5a46c4a2e Oct 30 20:46:53.782825 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.782796 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:46:53.782920 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.782859 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:46:53.782920 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.782888 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:46:53.785911 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:53.785890 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:54.049569 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:54.049489 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" event={"ID":"fd9df289-7dc4-4c00-9b75-d57dd17656f4","Type":"ContainerStarted","Data":"98a23860368255afe482d86497b28db63aec70939576c01430f3948ecf539e73"} Oct 30 20:46:54.049569 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:54.049551 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" event={"ID":"fd9df289-7dc4-4c00-9b75-d57dd17656f4","Type":"ContainerStarted","Data":"dc06c2aadd6142885d68c29fb0aba6b46bc011827d3bb59968cfdda5a46c4a2e"} Oct 30 20:46:54.066564 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:54.066419 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9"] Oct 30 20:46:54.068689 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:46:54.068657 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a2a74d1_f1a1_4434_bc2f_966fb5123180.slice/crio-2f424d8dff7547baa085a9fc29c3cbe9dd57515af500aad19f113a0bcf01bba7 WatchSource:0}: Error finding container 2f424d8dff7547baa085a9fc29c3cbe9dd57515af500aad19f113a0bcf01bba7: Status 404 returned error can't find the container with id 2f424d8dff7547baa085a9fc29c3cbe9dd57515af500aad19f113a0bcf01bba7 Oct 30 20:46:54.071356 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:54.071317 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:46:54.071470 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:54.071399 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:46:54.071470 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:54.071439 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:46:54.072931 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:54.072890 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" podStartSLOduration=1.072871258 podStartE2EDuration="1.072871258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:54.070227716 +0000 UTC m=+519.250124216" watchObservedRunningTime="2026-10-30 20:46:54.072871258 +0000 UTC m=+519.252767767" Oct 30 20:46:54.492231 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:54.492190 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:54.497324 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:54.497300 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:55.055821 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:55.055749 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" event={"ID":"2a2a74d1-f1a1-4434-bc2f-966fb5123180","Type":"ContainerStarted","Data":"0e16f4acf08a5799830c7ce55f9c97d6bed4bb555500535d3a2b156c5ea42e15"} Oct 30 20:46:55.056280 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:55.055837 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:55.056280 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:55.055881 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" event={"ID":"2a2a74d1-f1a1-4434-bc2f-966fb5123180","Type":"ContainerStarted","Data":"2f424d8dff7547baa085a9fc29c3cbe9dd57515af500aad19f113a0bcf01bba7"} Oct 30 20:46:55.056790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:55.056756 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:46:55.079726 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:55.079576 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" podStartSLOduration=2.079557739 podStartE2EDuration="2.079557739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:55.079458716 +0000 UTC m=+520.259355222" watchObservedRunningTime="2026-10-30 20:46:55.079557739 +0000 UTC m=+520.259454247" Oct 30 20:46:55.786509 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:55.786474 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:55.793008 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:55.792980 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:56.066001 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:56.065914 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:56.066724 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:56.066704 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:46:56.874941 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:56.874909 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6"] Oct 30 20:46:56.878437 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:56.878419 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6" Oct 30 20:46:56.888746 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:56.888716 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b612c7f2\"" Oct 30 20:46:56.889469 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:56.889455 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b612c7f2-dockercfg-5pmh4\"" Oct 30 20:46:56.890792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:56.890743 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6"] Oct 30 20:46:56.912427 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:56.912400 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml"] Oct 30 20:46:56.915882 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:56.915866 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml" Oct 30 20:46:56.933143 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:56.933120 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml"] Oct 30 20:46:56.953501 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:56.953471 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vcjm\" (UniqueName: \"kubernetes.io/projected/50840f14-a207-4145-b855-aea24f6fa722-kube-api-access-5vcjm\") pod \"payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml\" (UID: \"50840f14-a207-4145-b855-aea24f6fa722\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml" Oct 30 20:46:56.953668 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:56.953522 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb1d9de-4456-4aee-9968-27eaa4be7479-plugins-config-volume\") pod \"payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6\" (UID: \"3cb1d9de-4456-4aee-9968-27eaa4be7479\") " pod="openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6" Oct 30 20:46:56.953668 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:56.953618 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50840f14-a207-4145-b855-aea24f6fa722-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml\" (UID: \"50840f14-a207-4145-b855-aea24f6fa722\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml" Oct 30 20:46:56.953755 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:56.953736 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd89z\" (UniqueName: \"kubernetes.io/projected/3cb1d9de-4456-4aee-9968-27eaa4be7479-kube-api-access-hd89z\") pod \"payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6\" (UID: \"3cb1d9de-4456-4aee-9968-27eaa4be7479\") " pod="openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6" Oct 30 20:46:57.054561 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.054528 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd89z\" (UniqueName: \"kubernetes.io/projected/3cb1d9de-4456-4aee-9968-27eaa4be7479-kube-api-access-hd89z\") pod \"payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6\" (UID: \"3cb1d9de-4456-4aee-9968-27eaa4be7479\") " pod="openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6" Oct 30 20:46:57.054726 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.054580 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vcjm\" (UniqueName: \"kubernetes.io/projected/50840f14-a207-4145-b855-aea24f6fa722-kube-api-access-5vcjm\") pod \"payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml\" (UID: \"50840f14-a207-4145-b855-aea24f6fa722\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml" Oct 30 20:46:57.054726 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.054625 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb1d9de-4456-4aee-9968-27eaa4be7479-plugins-config-volume\") pod \"payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6\" (UID: \"3cb1d9de-4456-4aee-9968-27eaa4be7479\") " pod="openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6" Oct 30 20:46:57.054726 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.054661 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50840f14-a207-4145-b855-aea24f6fa722-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml\" (UID: \"50840f14-a207-4145-b855-aea24f6fa722\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml" Oct 30 20:46:57.055385 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.055364 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50840f14-a207-4145-b855-aea24f6fa722-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml\" (UID: \"50840f14-a207-4145-b855-aea24f6fa722\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml" Oct 30 20:46:57.055445 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.055425 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb1d9de-4456-4aee-9968-27eaa4be7479-plugins-config-volume\") pod \"payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6\" (UID: \"3cb1d9de-4456-4aee-9968-27eaa4be7479\") " pod="openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6" Oct 30 20:46:57.063424 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.063397 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd89z\" (UniqueName: \"kubernetes.io/projected/3cb1d9de-4456-4aee-9968-27eaa4be7479-kube-api-access-hd89z\") pod \"payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6\" (UID: \"3cb1d9de-4456-4aee-9968-27eaa4be7479\") " pod="openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6" Oct 30 20:46:57.063570 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.063554 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vcjm\" (UniqueName: \"kubernetes.io/projected/50840f14-a207-4145-b855-aea24f6fa722-kube-api-access-5vcjm\") pod \"payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml\" (UID: \"50840f14-a207-4145-b855-aea24f6fa722\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml" Oct 30 20:46:57.191861 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.191822 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6" Oct 30 20:46:57.232177 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.232133 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml" Oct 30 20:46:57.472413 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.472376 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6"] Oct 30 20:46:57.473130 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:46:57.473102 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb1d9de_4456_4aee_9968_27eaa4be7479.slice/crio-d827594dac25c2536941ad74fdbc46eb817d453d1cbad19995c1521731bd9a97 WatchSource:0}: Error finding container d827594dac25c2536941ad74fdbc46eb817d453d1cbad19995c1521731bd9a97: Status 404 returned error can't find the container with id d827594dac25c2536941ad74fdbc46eb817d453d1cbad19995c1521731bd9a97 Oct 30 20:46:57.548177 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:46:57.548141 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50840f14_a207_4145_b855_aea24f6fa722.slice/crio-f8aac83b9edb7f07cfbf8ce5fced5929a1baccb399c20cfded0ac61d449236ed WatchSource:0}: Error finding container f8aac83b9edb7f07cfbf8ce5fced5929a1baccb399c20cfded0ac61d449236ed: Status 404 returned error can't find the container with id f8aac83b9edb7f07cfbf8ce5fced5929a1baccb399c20cfded0ac61d449236ed Oct 30 20:46:57.548658 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.548636 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml"] Oct 30 20:46:57.593537 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.593510 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g"] Oct 30 20:46:57.598319 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.598294 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g" Oct 30 20:46:57.607102 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.607072 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-67598f1d\"" Oct 30 20:46:57.609515 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.608777 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g"] Oct 30 20:46:57.660853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.660817 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91767111-f078-4f24-88dd-fd4ee52fd113-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g\" (UID: \"91767111-f078-4f24-88dd-fd4ee52fd113\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g" Oct 30 20:46:57.661022 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.660858 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wcvl\" (UniqueName: \"kubernetes.io/projected/91767111-f078-4f24-88dd-fd4ee52fd113-kube-api-access-6wcvl\") pod \"payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g\" (UID: \"91767111-f078-4f24-88dd-fd4ee52fd113\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g" Oct 30 20:46:57.762038 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.761954 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91767111-f078-4f24-88dd-fd4ee52fd113-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g\" (UID: \"91767111-f078-4f24-88dd-fd4ee52fd113\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g" Oct 30 20:46:57.762038 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.762015 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wcvl\" (UniqueName: \"kubernetes.io/projected/91767111-f078-4f24-88dd-fd4ee52fd113-kube-api-access-6wcvl\") pod \"payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g\" (UID: \"91767111-f078-4f24-88dd-fd4ee52fd113\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g" Oct 30 20:46:57.762988 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.762951 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91767111-f078-4f24-88dd-fd4ee52fd113-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g\" (UID: \"91767111-f078-4f24-88dd-fd4ee52fd113\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g" Oct 30 20:46:57.771363 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.771338 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wcvl\" (UniqueName: \"kubernetes.io/projected/91767111-f078-4f24-88dd-fd4ee52fd113-kube-api-access-6wcvl\") pod \"payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g\" (UID: \"91767111-f078-4f24-88dd-fd4ee52fd113\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g" Oct 30 20:46:57.873558 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.873529 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68"] Oct 30 20:46:57.877448 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.877428 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" Oct 30 20:46:57.888577 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.888544 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68"] Oct 30 20:46:57.889841 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.889583 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b612c7f2-serving-cert\"" Oct 30 20:46:57.889841 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.889744 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 20:46:57.890231 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.890198 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bbd9l\"" Oct 30 20:46:57.913336 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.913310 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g" Oct 30 20:46:57.963736 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.963704 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-openshift-service-ca\") pod \"maas-api-e2e-ait-b612c7f2-865c9754c4-qff68\" (UID: \"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" Oct 30 20:46:57.963927 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.963903 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-metrics-certs\") pod \"maas-api-e2e-ait-b612c7f2-865c9754c4-qff68\" (UID: \"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" Oct 30 20:46:57.964065 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.964037 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-maas-api-tls\") pod \"maas-api-e2e-ait-b612c7f2-865c9754c4-qff68\" (UID: \"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" Oct 30 20:46:57.964243 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:57.964077 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks8xk\" (UniqueName: \"kubernetes.io/projected/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-kube-api-access-ks8xk\") pod \"maas-api-e2e-ait-b612c7f2-865c9754c4-qff68\" (UID: \"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" Oct 30 20:46:58.071699 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:58.071074 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-maas-api-tls\") pod \"maas-api-e2e-ait-b612c7f2-865c9754c4-qff68\" (UID: \"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" Oct 30 20:46:58.071699 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:58.071125 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks8xk\" (UniqueName: \"kubernetes.io/projected/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-kube-api-access-ks8xk\") pod \"maas-api-e2e-ait-b612c7f2-865c9754c4-qff68\" (UID: \"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" Oct 30 20:46:58.071699 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:58.071200 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-openshift-service-ca\") pod \"maas-api-e2e-ait-b612c7f2-865c9754c4-qff68\" (UID: \"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" Oct 30 20:46:58.071699 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:58.071225 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-metrics-certs\") pod \"maas-api-e2e-ait-b612c7f2-865c9754c4-qff68\" (UID: \"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" Oct 30 20:46:58.072296 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:58.072274 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-openshift-service-ca\") pod \"maas-api-e2e-ait-b612c7f2-865c9754c4-qff68\" (UID: \"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" Oct 30 20:46:58.076444 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:58.076417 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-metrics-certs\") pod \"maas-api-e2e-ait-b612c7f2-865c9754c4-qff68\" (UID: \"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" Oct 30 20:46:58.076858 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:58.076832 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-maas-api-tls\") pod \"maas-api-e2e-ait-b612c7f2-865c9754c4-qff68\" (UID: \"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" Oct 30 20:46:58.080121 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:58.080092 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6" event={"ID":"3cb1d9de-4456-4aee-9968-27eaa4be7479","Type":"ContainerStarted","Data":"8571aaba7a867c7304724b8c1cab66cbc1e00741e3979e860e91606218190d76"} Oct 30 20:46:58.080255 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:58.080156 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6" event={"ID":"3cb1d9de-4456-4aee-9968-27eaa4be7479","Type":"ContainerStarted","Data":"d827594dac25c2536941ad74fdbc46eb817d453d1cbad19995c1521731bd9a97"} Oct 30 20:46:58.080255 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:58.080201 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6" Oct 30 20:46:58.081900 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:58.081876 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml" event={"ID":"50840f14-a207-4145-b855-aea24f6fa722","Type":"ContainerStarted","Data":"90ff0b1c69e0c9acb924e368b314aca095d2e424b09d78a0b87110c5400daa2a"} Oct 30 20:46:58.082008 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:58.081922 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml" event={"ID":"50840f14-a207-4145-b855-aea24f6fa722","Type":"ContainerStarted","Data":"f8aac83b9edb7f07cfbf8ce5fced5929a1baccb399c20cfded0ac61d449236ed"} Oct 30 20:46:58.082008 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:58.081993 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml" Oct 30 20:46:58.087498 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:58.087464 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks8xk\" (UniqueName: \"kubernetes.io/projected/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-kube-api-access-ks8xk\") pod \"maas-api-e2e-ait-b612c7f2-865c9754c4-qff68\" (UID: \"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" Oct 30 20:46:58.105950 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:58.105893 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6" podStartSLOduration=2.105876826 podStartE2EDuration="2.105876826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:58.102295523 +0000 UTC m=+523.282192035" watchObservedRunningTime="2026-10-30 20:46:58.105876826 +0000 UTC m=+523.285773334" Oct 30 20:46:58.137353 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:58.137307 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml" podStartSLOduration=2.137291593 podStartE2EDuration="2.137291593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:58.135617791 +0000 UTC m=+523.315514299" watchObservedRunningTime="2026-10-30 20:46:58.137291593 +0000 UTC m=+523.317188101" Oct 30 20:46:58.192881 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:58.192849 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" Oct 30 20:46:58.193286 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:58.193268 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g"] Oct 30 20:46:58.197153 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:46:58.197123 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91767111_f078_4f24_88dd_fd4ee52fd113.slice/crio-81b0b364e44ed66089277b435ce78f2b78d117565e92972df7f84169626f86f2 WatchSource:0}: Error finding container 81b0b364e44ed66089277b435ce78f2b78d117565e92972df7f84169626f86f2: Status 404 returned error can't find the container with id 81b0b364e44ed66089277b435ce78f2b78d117565e92972df7f84169626f86f2 Oct 30 20:46:58.484688 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:58.484645 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68"] Oct 30 20:46:58.487433 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:46:58.487405 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c875fb6_8fab_46ee_8d8c_5adb3d8b5e1e.slice/crio-49773510af03abe960d60ea3e0f48cd5c323fb6afb184ee62018ede5822bd496 WatchSource:0}: Error finding container 49773510af03abe960d60ea3e0f48cd5c323fb6afb184ee62018ede5822bd496: Status 404 returned error can't find the container with id 49773510af03abe960d60ea3e0f48cd5c323fb6afb184ee62018ede5822bd496 Oct 30 20:46:59.090843 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:59.090804 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" event={"ID":"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e","Type":"ContainerStarted","Data":"cfd30cd4e16c0f36db5041b437395c73877555899e14fd0d4468daafa13a5309"} Oct 30 20:46:59.091046 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:59.090873 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" event={"ID":"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e","Type":"ContainerStarted","Data":"49773510af03abe960d60ea3e0f48cd5c323fb6afb184ee62018ede5822bd496"} Oct 30 20:46:59.091046 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:59.090943 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" Oct 30 20:46:59.092750 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:59.092722 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g" event={"ID":"91767111-f078-4f24-88dd-fd4ee52fd113","Type":"ContainerStarted","Data":"7f81c6a4831383c026202f7956cf17e0950a911225407b1c268ab2125dfd4943"} Oct 30 20:46:59.092923 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:59.092818 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g" Oct 30 20:46:59.092923 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:59.092844 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g" event={"ID":"91767111-f078-4f24-88dd-fd4ee52fd113","Type":"ContainerStarted","Data":"81b0b364e44ed66089277b435ce78f2b78d117565e92972df7f84169626f86f2"} Oct 30 20:46:59.114989 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:59.114934 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" podStartSLOduration=1.602113831 podStartE2EDuration="2.114915021s" totalImagesPullingTime="512.80119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:59.113443514 +0000 UTC m=+524.293340023" watchObservedRunningTime="2026-10-30 20:46:59.114915021 +0000 UTC m=+524.294811528" Oct 30 20:46:59.137493 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:59.137448 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g" podStartSLOduration=2.137433909 podStartE2EDuration="2.137433909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:46:59.135686606 +0000 UTC m=+524.315583114" watchObservedRunningTime="2026-10-30 20:46:59.137433909 +0000 UTC m=+524.317330416" Oct 30 20:46:59.620029 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:59.619997 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5"] Oct 30 20:46:59.624203 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:59.624181 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5" Oct 30 20:46:59.634421 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:59.634402 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5"] Oct 30 20:46:59.635382 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:59.635356 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b16c21be-dockercfg-nk48k\"" Oct 30 20:46:59.691116 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:59.691088 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f1aae51-6213-467c-a0ef-26f5fca2bdef-plugins-config-volume\") pod \"payload-processing-e2e-ait-b16c21be-c777df947-zkld5\" (UID: \"1f1aae51-6213-467c-a0ef-26f5fca2bdef\") " pod="openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5" Oct 30 20:46:59.691282 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:59.691126 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlfr8\" (UniqueName: \"kubernetes.io/projected/1f1aae51-6213-467c-a0ef-26f5fca2bdef-kube-api-access-wlfr8\") pod \"payload-processing-e2e-ait-b16c21be-c777df947-zkld5\" (UID: \"1f1aae51-6213-467c-a0ef-26f5fca2bdef\") " pod="openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5" Oct 30 20:46:59.718076 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:59.718048 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b16c21be\"" Oct 30 20:46:59.792537 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:59.792494 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f1aae51-6213-467c-a0ef-26f5fca2bdef-plugins-config-volume\") pod \"payload-processing-e2e-ait-b16c21be-c777df947-zkld5\" (UID: \"1f1aae51-6213-467c-a0ef-26f5fca2bdef\") " pod="openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5" Oct 30 20:46:59.792684 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:59.792549 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlfr8\" (UniqueName: \"kubernetes.io/projected/1f1aae51-6213-467c-a0ef-26f5fca2bdef-kube-api-access-wlfr8\") pod \"payload-processing-e2e-ait-b16c21be-c777df947-zkld5\" (UID: \"1f1aae51-6213-467c-a0ef-26f5fca2bdef\") " pod="openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5" Oct 30 20:46:59.793210 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:59.793191 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f1aae51-6213-467c-a0ef-26f5fca2bdef-plugins-config-volume\") pod \"payload-processing-e2e-ait-b16c21be-c777df947-zkld5\" (UID: \"1f1aae51-6213-467c-a0ef-26f5fca2bdef\") " pod="openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5" Oct 30 20:46:59.802265 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:59.802243 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlfr8\" (UniqueName: \"kubernetes.io/projected/1f1aae51-6213-467c-a0ef-26f5fca2bdef-kube-api-access-wlfr8\") pod \"payload-processing-e2e-ait-b16c21be-c777df947-zkld5\" (UID: \"1f1aae51-6213-467c-a0ef-26f5fca2bdef\") " pod="openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5" Oct 30 20:46:59.939643 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:46:59.939611 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5" Oct 30 20:47:00.189808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:00.189719 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz"] Oct 30 20:47:00.195137 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:00.195112 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" Oct 30 20:47:00.204718 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:00.204693 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz"] Oct 30 20:47:00.219322 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:00.219299 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5"] Oct 30 20:47:00.221595 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:47:00.221561 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1aae51_6213_467c_a0ef_26f5fca2bdef.slice/crio-41a7324393364c20037b35a9d457fae829b4431529e42a7c3bfce87d38c03572 WatchSource:0}: Error finding container 41a7324393364c20037b35a9d457fae829b4431529e42a7c3bfce87d38c03572: Status 404 returned error can't find the container with id 41a7324393364c20037b35a9d457fae829b4431529e42a7c3bfce87d38c03572 Oct 30 20:47:00.238663 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:00.238640 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-67598f1d-serving-cert\"" Oct 30 20:47:00.298600 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:00.298568 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39bb7b01-c262-4d9d-b919-74156bff3b41-openshift-service-ca\") pod \"maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz\" (UID: \"39bb7b01-c262-4d9d-b919-74156bff3b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" Oct 30 20:47:00.298723 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:00.298702 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39bb7b01-c262-4d9d-b919-74156bff3b41-maas-api-tls\") pod \"maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz\" (UID: \"39bb7b01-c262-4d9d-b919-74156bff3b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" Oct 30 20:47:00.298801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:00.298736 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lk9c\" (UniqueName: \"kubernetes.io/projected/39bb7b01-c262-4d9d-b919-74156bff3b41-kube-api-access-5lk9c\") pod \"maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz\" (UID: \"39bb7b01-c262-4d9d-b919-74156bff3b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" Oct 30 20:47:00.298921 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:00.298895 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39bb7b01-c262-4d9d-b919-74156bff3b41-metrics-certs\") pod \"maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz\" (UID: \"39bb7b01-c262-4d9d-b919-74156bff3b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" Oct 30 20:47:00.400147 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:00.400111 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39bb7b01-c262-4d9d-b919-74156bff3b41-maas-api-tls\") pod \"maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz\" (UID: \"39bb7b01-c262-4d9d-b919-74156bff3b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" Oct 30 20:47:00.400147 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:00.400151 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lk9c\" (UniqueName: \"kubernetes.io/projected/39bb7b01-c262-4d9d-b919-74156bff3b41-kube-api-access-5lk9c\") pod \"maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz\" (UID: \"39bb7b01-c262-4d9d-b919-74156bff3b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" Oct 30 20:47:00.400371 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:00.400278 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39bb7b01-c262-4d9d-b919-74156bff3b41-metrics-certs\") pod \"maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz\" (UID: \"39bb7b01-c262-4d9d-b919-74156bff3b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" Oct 30 20:47:00.400422 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:00.400382 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39bb7b01-c262-4d9d-b919-74156bff3b41-openshift-service-ca\") pod \"maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz\" (UID: \"39bb7b01-c262-4d9d-b919-74156bff3b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" Oct 30 20:47:00.401047 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:00.401025 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39bb7b01-c262-4d9d-b919-74156bff3b41-openshift-service-ca\") pod \"maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz\" (UID: \"39bb7b01-c262-4d9d-b919-74156bff3b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" Oct 30 20:47:00.402704 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:00.402683 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39bb7b01-c262-4d9d-b919-74156bff3b41-metrics-certs\") pod \"maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz\" (UID: \"39bb7b01-c262-4d9d-b919-74156bff3b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" Oct 30 20:47:00.402824 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:00.402757 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39bb7b01-c262-4d9d-b919-74156bff3b41-maas-api-tls\") pod \"maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz\" (UID: \"39bb7b01-c262-4d9d-b919-74156bff3b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" Oct 30 20:47:00.409199 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:00.409178 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lk9c\" (UniqueName: \"kubernetes.io/projected/39bb7b01-c262-4d9d-b919-74156bff3b41-kube-api-access-5lk9c\") pod \"maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz\" (UID: \"39bb7b01-c262-4d9d-b919-74156bff3b41\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" Oct 30 20:47:00.512587 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:00.512513 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" Oct 30 20:47:00.790592 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:00.790567 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz"] Oct 30 20:47:00.792411 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:47:00.792377 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39bb7b01_c262_4d9d_b919_74156bff3b41.slice/crio-87dcd2cbc04b6a26badf04b234b56eda3535729c85d73da620c04f19f726cdfc WatchSource:0}: Error finding container 87dcd2cbc04b6a26badf04b234b56eda3535729c85d73da620c04f19f726cdfc: Status 404 returned error can't find the container with id 87dcd2cbc04b6a26badf04b234b56eda3535729c85d73da620c04f19f726cdfc Oct 30 20:47:01.103552 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.103469 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5" event={"ID":"1f1aae51-6213-467c-a0ef-26f5fca2bdef","Type":"ContainerStarted","Data":"18f62328bb0589c1343c34783610e7ff898b5c06b7da262a0bfbe3335c9821a2"} Oct 30 20:47:01.103552 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.103530 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5" Oct 30 20:47:01.103552 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.103550 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5" event={"ID":"1f1aae51-6213-467c-a0ef-26f5fca2bdef","Type":"ContainerStarted","Data":"41a7324393364c20037b35a9d457fae829b4431529e42a7c3bfce87d38c03572"} Oct 30 20:47:01.104927 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.104901 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" event={"ID":"39bb7b01-c262-4d9d-b919-74156bff3b41","Type":"ContainerStarted","Data":"87dcd2cbc04b6a26badf04b234b56eda3535729c85d73da620c04f19f726cdfc"} Oct 30 20:47:01.122219 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.122171 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5" podStartSLOduration=2.122160268 podStartE2EDuration="2.122160268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:01.12057554 +0000 UTC m=+526.300472050" watchObservedRunningTime="2026-10-30 20:47:01.122160268 +0000 UTC m=+526.302056776" Oct 30 20:47:01.208687 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.208560 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq"] Oct 30 20:47:01.257921 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.257886 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" Oct 30 20:47:01.258360 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.258088 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq"] Oct 30 20:47:01.263924 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.263902 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-8f46c771-serving-cert\"" Oct 30 20:47:01.328034 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.309920 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59c21318-1027-43ea-8f08-7f67b4d097ed-openshift-service-ca\") pod \"maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq\" (UID: \"59c21318-1027-43ea-8f08-7f67b4d097ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" Oct 30 20:47:01.328034 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.309964 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59c21318-1027-43ea-8f08-7f67b4d097ed-maas-api-tls\") pod \"maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq\" (UID: \"59c21318-1027-43ea-8f08-7f67b4d097ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" Oct 30 20:47:01.328034 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.310031 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59c21318-1027-43ea-8f08-7f67b4d097ed-metrics-certs\") pod \"maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq\" (UID: \"59c21318-1027-43ea-8f08-7f67b4d097ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" Oct 30 20:47:01.328034 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.310065 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29ccr\" (UniqueName: \"kubernetes.io/projected/59c21318-1027-43ea-8f08-7f67b4d097ed-kube-api-access-29ccr\") pod \"maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq\" (UID: \"59c21318-1027-43ea-8f08-7f67b4d097ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" Oct 30 20:47:01.426841 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.423384 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59c21318-1027-43ea-8f08-7f67b4d097ed-metrics-certs\") pod \"maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq\" (UID: \"59c21318-1027-43ea-8f08-7f67b4d097ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" Oct 30 20:47:01.426841 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.423447 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29ccr\" (UniqueName: \"kubernetes.io/projected/59c21318-1027-43ea-8f08-7f67b4d097ed-kube-api-access-29ccr\") pod \"maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq\" (UID: \"59c21318-1027-43ea-8f08-7f67b4d097ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" Oct 30 20:47:01.426841 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.423599 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59c21318-1027-43ea-8f08-7f67b4d097ed-openshift-service-ca\") pod \"maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq\" (UID: \"59c21318-1027-43ea-8f08-7f67b4d097ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" Oct 30 20:47:01.426841 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.423628 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59c21318-1027-43ea-8f08-7f67b4d097ed-maas-api-tls\") pod \"maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq\" (UID: \"59c21318-1027-43ea-8f08-7f67b4d097ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" Oct 30 20:47:01.429952 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.429826 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59c21318-1027-43ea-8f08-7f67b4d097ed-openshift-service-ca\") pod \"maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq\" (UID: \"59c21318-1027-43ea-8f08-7f67b4d097ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" Oct 30 20:47:01.431283 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.431253 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59c21318-1027-43ea-8f08-7f67b4d097ed-maas-api-tls\") pod \"maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq\" (UID: \"59c21318-1027-43ea-8f08-7f67b4d097ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" Oct 30 20:47:01.431426 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.431259 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59c21318-1027-43ea-8f08-7f67b4d097ed-metrics-certs\") pod \"maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq\" (UID: \"59c21318-1027-43ea-8f08-7f67b4d097ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" Oct 30 20:47:01.454750 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.454717 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29ccr\" (UniqueName: \"kubernetes.io/projected/59c21318-1027-43ea-8f08-7f67b4d097ed-kube-api-access-29ccr\") pod \"maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq\" (UID: \"59c21318-1027-43ea-8f08-7f67b4d097ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" Oct 30 20:47:01.574098 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.574063 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" Oct 30 20:47:01.873650 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:01.873578 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq"] Oct 30 20:47:01.884494 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:47:01.884460 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c21318_1027_43ea_8f08_7f67b4d097ed.slice/crio-12216d8ccb12f5233638dfd26805db320728ffb92470d10f9c6a8a1a322c5aef WatchSource:0}: Error finding container 12216d8ccb12f5233638dfd26805db320728ffb92470d10f9c6a8a1a322c5aef: Status 404 returned error can't find the container with id 12216d8ccb12f5233638dfd26805db320728ffb92470d10f9c6a8a1a322c5aef Oct 30 20:47:02.133129 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:02.133039 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" event={"ID":"39bb7b01-c262-4d9d-b919-74156bff3b41","Type":"ContainerStarted","Data":"fb00f5792fd61211a8cfeeb5b75ee3238b5bbe3e6815fd5650f16cd5fa0cbca6"} Oct 30 20:47:02.133807 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:02.133756 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" Oct 30 20:47:02.134400 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:02.134376 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" event={"ID":"59c21318-1027-43ea-8f08-7f67b4d097ed","Type":"ContainerStarted","Data":"12216d8ccb12f5233638dfd26805db320728ffb92470d10f9c6a8a1a322c5aef"} Oct 30 20:47:02.160788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:02.155970 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" podStartSLOduration=1.701768264 podStartE2EDuration="2.155951399s" totalImagesPullingTime="454.183135ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:02.153904156 +0000 UTC m=+527.333800660" watchObservedRunningTime="2026-10-30 20:47:02.155951399 +0000 UTC m=+527.335847909" Oct 30 20:47:03.140893 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:03.140858 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" event={"ID":"59c21318-1027-43ea-8f08-7f67b4d097ed","Type":"ContainerStarted","Data":"3e97b85d4afc89d7ff5d97395e663d9cb8bcdca8a551bcc13dceeac00f55e2a1"} Oct 30 20:47:03.141353 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:03.141126 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" Oct 30 20:47:03.161807 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:03.161742 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" podStartSLOduration=1.6999781619999998 podStartE2EDuration="2.161729422s" totalImagesPullingTime="461.75126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:03.159852082 +0000 UTC m=+528.339748591" watchObservedRunningTime="2026-10-30 20:47:03.161729422 +0000 UTC m=+528.341625928" Oct 30 20:47:05.102414 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:05.102379 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" Oct 30 20:47:09.096674 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:09.096640 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml" Oct 30 20:47:09.097193 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:09.096720 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6" Oct 30 20:47:09.163400 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:09.163368 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" Oct 30 20:47:09.164294 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:09.164267 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" Oct 30 20:47:10.101141 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:10.100888 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g" Oct 30 20:47:11.090423 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.089793 4848 patch_prober.go:29] interesting pod/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:47:11.090423 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.089856 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" podUID="fd9df289-7dc4-4c00-9b75-d57dd17656f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:11.233829 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.233789 4848 generic.go:356] "Generic (PLEG): container finished" podID="fd9df289-7dc4-4c00-9b75-d57dd17656f4" containerID="98a23860368255afe482d86497b28db63aec70939576c01430f3948ecf539e73" exitCode=0 Oct 30 20:47:11.234332 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.233874 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" event={"ID":"fd9df289-7dc4-4c00-9b75-d57dd17656f4","Type":"ContainerDied","Data":"98a23860368255afe482d86497b28db63aec70939576c01430f3948ecf539e73"} Oct 30 20:47:11.234409 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.234359 4848 scope.go:122] "RemoveContainer" containerID="98a23860368255afe482d86497b28db63aec70939576c01430f3948ecf539e73" Oct 30 20:47:11.235803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.235747 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:11.235884 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.235856 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:11.235934 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.235920 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:11.291619 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.284918 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld"] Oct 30 20:47:11.291619 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.290662 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" Oct 30 20:47:11.301296 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.300273 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-eaf151e7-serving-cert\"" Oct 30 20:47:11.310990 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.302906 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld"] Oct 30 20:47:11.382351 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.382310 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh8jw\" (UniqueName: \"kubernetes.io/projected/84be7e6d-711c-46f8-a188-0f8955560e5a-kube-api-access-nh8jw\") pod \"maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld\" (UID: \"84be7e6d-711c-46f8-a188-0f8955560e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" Oct 30 20:47:11.382551 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.382392 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84be7e6d-711c-46f8-a188-0f8955560e5a-openshift-service-ca\") pod \"maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld\" (UID: \"84be7e6d-711c-46f8-a188-0f8955560e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" Oct 30 20:47:11.382640 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.382610 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84be7e6d-711c-46f8-a188-0f8955560e5a-metrics-certs\") pod \"maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld\" (UID: \"84be7e6d-711c-46f8-a188-0f8955560e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" Oct 30 20:47:11.382722 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.382663 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84be7e6d-711c-46f8-a188-0f8955560e5a-maas-api-tls\") pod \"maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld\" (UID: \"84be7e6d-711c-46f8-a188-0f8955560e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" Oct 30 20:47:11.483905 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.483796 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84be7e6d-711c-46f8-a188-0f8955560e5a-metrics-certs\") pod \"maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld\" (UID: \"84be7e6d-711c-46f8-a188-0f8955560e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" Oct 30 20:47:11.483905 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.483879 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84be7e6d-711c-46f8-a188-0f8955560e5a-maas-api-tls\") pod \"maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld\" (UID: \"84be7e6d-711c-46f8-a188-0f8955560e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" Oct 30 20:47:11.484224 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.484000 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh8jw\" (UniqueName: \"kubernetes.io/projected/84be7e6d-711c-46f8-a188-0f8955560e5a-kube-api-access-nh8jw\") pod \"maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld\" (UID: \"84be7e6d-711c-46f8-a188-0f8955560e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" Oct 30 20:47:11.484224 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:47:11.484021 4848 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-serving-cert: secret "maas-api-e2e-derive-eaf151e7-serving-cert" not found Oct 30 20:47:11.484224 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.484056 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84be7e6d-711c-46f8-a188-0f8955560e5a-openshift-service-ca\") pod \"maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld\" (UID: \"84be7e6d-711c-46f8-a188-0f8955560e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" Oct 30 20:47:11.484224 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:47:11.484114 4848 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84be7e6d-711c-46f8-a188-0f8955560e5a-maas-api-tls podName:84be7e6d-711c-46f8-a188-0f8955560e5a nodeName:}" failed. No retries permitted until 2026-10-30 20:47:11.984089212 +0000 UTC m=+537.163985718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/84be7e6d-711c-46f8-a188-0f8955560e5a-maas-api-tls") pod "maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" (UID: "84be7e6d-711c-46f8-a188-0f8955560e5a") : secret "maas-api-e2e-derive-eaf151e7-serving-cert" not found Oct 30 20:47:11.484936 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.484896 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84be7e6d-711c-46f8-a188-0f8955560e5a-openshift-service-ca\") pod \"maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld\" (UID: \"84be7e6d-711c-46f8-a188-0f8955560e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" Oct 30 20:47:11.486860 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.486832 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84be7e6d-711c-46f8-a188-0f8955560e5a-metrics-certs\") pod \"maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld\" (UID: \"84be7e6d-711c-46f8-a188-0f8955560e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" Oct 30 20:47:11.496312 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.496279 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh8jw\" (UniqueName: \"kubernetes.io/projected/84be7e6d-711c-46f8-a188-0f8955560e5a-kube-api-access-nh8jw\") pod \"maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld\" (UID: \"84be7e6d-711c-46f8-a188-0f8955560e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" Oct 30 20:47:11.989851 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.989800 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84be7e6d-711c-46f8-a188-0f8955560e5a-maas-api-tls\") pod \"maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld\" (UID: \"84be7e6d-711c-46f8-a188-0f8955560e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" Oct 30 20:47:11.992801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:11.992743 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84be7e6d-711c-46f8-a188-0f8955560e5a-maas-api-tls\") pod \"maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld\" (UID: \"84be7e6d-711c-46f8-a188-0f8955560e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" Oct 30 20:47:12.115795 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:12.109889 4848 patch_prober.go:29] interesting pod/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:47:12.115795 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:12.109959 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" podUID="2a2a74d1-f1a1-4434-bc2f-966fb5123180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:12.137881 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:12.137852 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5" Oct 30 20:47:12.232789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:12.228960 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" Oct 30 20:47:12.247561 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:12.247123 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" event={"ID":"fd9df289-7dc4-4c00-9b75-d57dd17656f4","Type":"ContainerStarted","Data":"fe0ea8cb3666cc5ace7a83c24d313efc9d19cae60ffc5cdaeef6669f454f1265"} Oct 30 20:47:12.492099 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:12.491985 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:47:12.492369 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:12.492343 4848 patch_prober.go:29] interesting pod/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:47:12.492483 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:12.492405 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" podUID="fd9df289-7dc4-4c00-9b75-d57dd17656f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 20:47:12.557865 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:12.557835 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld"] Oct 30 20:47:12.559925 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:47:12.559898 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84be7e6d_711c_46f8_a188_0f8955560e5a.slice/crio-4f75a171a3bec5ccba1d2f284b771ae551435436d9c4217c9f00f234d5734276 WatchSource:0}: Error finding container 4f75a171a3bec5ccba1d2f284b771ae551435436d9c4217c9f00f234d5734276: Status 404 returned error can't find the container with id 4f75a171a3bec5ccba1d2f284b771ae551435436d9c4217c9f00f234d5734276 Oct 30 20:47:13.255122 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:13.255033 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" event={"ID":"84be7e6d-711c-46f8-a188-0f8955560e5a","Type":"ContainerStarted","Data":"e42ce637b917a6f0381b8b509426cd35c0a55b0c67af3c771239a94efec64213"} Oct 30 20:47:13.255122 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:13.255094 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" event={"ID":"84be7e6d-711c-46f8-a188-0f8955560e5a","Type":"ContainerStarted","Data":"4f75a171a3bec5ccba1d2f284b771ae551435436d9c4217c9f00f234d5734276"} Oct 30 20:47:13.255675 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:13.255275 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" Oct 30 20:47:13.283474 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:13.283414 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" podStartSLOduration=1.858161242 podStartE2EDuration="2.283395135s" totalImagesPullingTime="425.233893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:13.281087353 +0000 UTC m=+538.460983863" watchObservedRunningTime="2026-10-30 20:47:13.283395135 +0000 UTC m=+538.463291643" Oct 30 20:47:13.492678 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:13.492630 4848 patch_prober.go:29] interesting pod/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:47:13.492890 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:13.492687 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" podUID="fd9df289-7dc4-4c00-9b75-d57dd17656f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 20:47:14.491749 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:14.491709 4848 patch_prober.go:29] interesting pod/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:47:14.494075 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:14.491788 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" podUID="fd9df289-7dc4-4c00-9b75-d57dd17656f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 20:47:15.268522 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:15.268437 4848 generic.go:356] "Generic (PLEG): container finished" podID="2a2a74d1-f1a1-4434-bc2f-966fb5123180" containerID="0e16f4acf08a5799830c7ce55f9c97d6bed4bb555500535d3a2b156c5ea42e15" exitCode=0 Oct 30 20:47:15.268522 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:15.268484 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" event={"ID":"2a2a74d1-f1a1-4434-bc2f-966fb5123180","Type":"ContainerDied","Data":"0e16f4acf08a5799830c7ce55f9c97d6bed4bb555500535d3a2b156c5ea42e15"} Oct 30 20:47:15.268854 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:15.268839 4848 scope.go:122] "RemoveContainer" containerID="0e16f4acf08a5799830c7ce55f9c97d6bed4bb555500535d3a2b156c5ea42e15" Oct 30 20:47:15.269988 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:15.269954 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:15.270108 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:15.270036 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:15.270108 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:15.270086 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:15.491818 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:15.491780 4848 patch_prober.go:29] interesting pod/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:47:15.492305 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:15.491838 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" podUID="fd9df289-7dc4-4c00-9b75-d57dd17656f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 20:47:16.275108 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:16.275075 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" event={"ID":"2a2a74d1-f1a1-4434-bc2f-966fb5123180","Type":"ContainerStarted","Data":"9cc32722a334548cd9111c857749b19b60c3787016a7f936a858115b7722950b"} Oct 30 20:47:16.491884 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:16.491849 4848 patch_prober.go:29] interesting pod/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 20:47:16.492368 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:16.491917 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" podUID="fd9df289-7dc4-4c00-9b75-d57dd17656f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 20:47:16.786861 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:16.786821 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:47:16.787285 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:16.787261 4848 patch_prober.go:29] interesting pod/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:47:16.787414 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:16.787315 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" podUID="2a2a74d1-f1a1-4434-bc2f-966fb5123180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 20:47:17.651744 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:17.651711 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:47:17.652213 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:17.652091 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:47:17.653081 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:17.653060 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:47:17.791388 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:17.790793 4848 patch_prober.go:29] interesting pod/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:47:17.791388 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:17.790878 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" podUID="2a2a74d1-f1a1-4434-bc2f-966fb5123180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 20:47:18.786338 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:18.786296 4848 patch_prober.go:29] interesting pod/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:47:18.786855 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:18.786353 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" podUID="2a2a74d1-f1a1-4434-bc2f-966fb5123180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 20:47:19.268289 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:19.268260 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" Oct 30 20:47:19.787261 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:19.787225 4848 patch_prober.go:29] interesting pod/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:47:19.787670 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:19.787298 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" podUID="2a2a74d1-f1a1-4434-bc2f-966fb5123180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 20:47:20.786167 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:20.786133 4848 patch_prober.go:29] interesting pod/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 20:47:20.786339 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:20.786196 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" podUID="2a2a74d1-f1a1-4434-bc2f-966fb5123180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 20:47:21.753980 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:21.753947 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6"] Oct 30 20:47:21.762126 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:21.762093 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml"] Oct 30 20:47:21.793146 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:21.793116 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:47:21.794038 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:21.794010 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:47:21.794364 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:21.794341 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:47:22.518056 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:47:22.518020 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:47:25.810698 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:25.810668 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68"] Oct 30 20:47:25.811251 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:25.810882 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" podUID="9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e" containerName="maas-api" containerID="cri-o://cfd30cd4e16c0f36db5041b437395c73877555899e14fd0d4468daafa13a5309" gracePeriod=30 Oct 30 20:47:26.090404 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.090352 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" Oct 30 20:47:26.275917 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.275886 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-openshift-service-ca\") pod \"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e\" (UID: \"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e\") " Oct 30 20:47:26.276096 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.275946 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-metrics-certs\") pod \"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e\" (UID: \"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e\") " Oct 30 20:47:26.276096 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.275990 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-maas-api-tls\") pod \"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e\" (UID: \"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e\") " Oct 30 20:47:26.276096 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.276019 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-kube-api-access-ks8xk\" (UniqueName: \"kubernetes.io/projected/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-kube-api-access-ks8xk\") pod \"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e\" (UID: \"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e\") " Oct 30 20:47:26.276423 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.276386 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-openshift-service-ca" pod "9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e" (UID: "9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:47:26.278251 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.278221 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-metrics-certs" pod "9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e" (UID: "9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:47:26.278343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.278291 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-kube-api-access-ks8xk" pod "9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e" (UID: "9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e"). InnerVolumeSpecName "kube-api-access-ks8xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:26.278343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.278305 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-maas-api-tls" pod "9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e" (UID: "9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:47:26.324222 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.324151 4848 generic.go:356] "Generic (PLEG): container finished" podID="9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e" containerID="cfd30cd4e16c0f36db5041b437395c73877555899e14fd0d4468daafa13a5309" exitCode=0 Oct 30 20:47:26.324222 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.324197 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" Oct 30 20:47:26.324406 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.324231 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" event={"ID":"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e","Type":"ContainerDied","Data":"cfd30cd4e16c0f36db5041b437395c73877555899e14fd0d4468daafa13a5309"} Oct 30 20:47:26.324406 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.324284 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68" event={"ID":"9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e","Type":"ContainerDied","Data":"49773510af03abe960d60ea3e0f48cd5c323fb6afb184ee62018ede5822bd496"} Oct 30 20:47:26.324406 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.324307 4848 scope.go:122] "RemoveContainer" containerID="cfd30cd4e16c0f36db5041b437395c73877555899e14fd0d4468daafa13a5309" Oct 30 20:47:26.333936 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.333912 4848 scope.go:122] "RemoveContainer" containerID="cfd30cd4e16c0f36db5041b437395c73877555899e14fd0d4468daafa13a5309" Oct 30 20:47:26.334228 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:47:26.334204 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd30cd4e16c0f36db5041b437395c73877555899e14fd0d4468daafa13a5309\": container with ID starting with cfd30cd4e16c0f36db5041b437395c73877555899e14fd0d4468daafa13a5309 not found: ID does not exist" containerID="cfd30cd4e16c0f36db5041b437395c73877555899e14fd0d4468daafa13a5309" Oct 30 20:47:26.334294 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.334241 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd30cd4e16c0f36db5041b437395c73877555899e14fd0d4468daafa13a5309"} err="failed to get container status \"cfd30cd4e16c0f36db5041b437395c73877555899e14fd0d4468daafa13a5309\": rpc error: code = NotFound desc = could not find container \"cfd30cd4e16c0f36db5041b437395c73877555899e14fd0d4468daafa13a5309\": container with ID starting with cfd30cd4e16c0f36db5041b437395c73877555899e14fd0d4468daafa13a5309 not found: ID does not exist" Oct 30 20:47:26.350659 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.350627 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68"] Oct 30 20:47:26.356448 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.356424 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b612c7f2-865c9754c4-qff68"] Oct 30 20:47:26.376810 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.376754 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:26.376810 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.376808 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:26.377010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.376820 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks8xk\" (UniqueName: \"kubernetes.io/projected/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-kube-api-access-ks8xk\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:26.377010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.376831 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:26.754931 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.754887 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6" podUID="3cb1d9de-4456-4aee-9968-27eaa4be7479" containerName="payload-processing" containerID="cri-o://8571aaba7a867c7304724b8c1cab66cbc1e00741e3979e860e91606218190d76" gracePeriod=25 Oct 30 20:47:26.762398 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.762331 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml" podUID="50840f14-a207-4145-b855-aea24f6fa722" containerName="payload-pre-processing" containerID="cri-o://90ff0b1c69e0c9acb924e368b314aca095d2e424b09d78a0b87110c5400daa2a" gracePeriod=25 Oct 30 20:47:26.939059 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:26.938957 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e" path="/var/lib/kubelet/pods/9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e/volumes" Oct 30 20:47:27.027906 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.027878 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6" Oct 30 20:47:27.057793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.057753 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml" Oct 30 20:47:27.186056 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.186028 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cb1d9de-4456-4aee-9968-27eaa4be7479-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb1d9de-4456-4aee-9968-27eaa4be7479-plugins-config-volume\") pod \"3cb1d9de-4456-4aee-9968-27eaa4be7479\" (UID: \"3cb1d9de-4456-4aee-9968-27eaa4be7479\") " Oct 30 20:47:27.186216 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.186068 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50840f14-a207-4145-b855-aea24f6fa722-kube-api-access-5vcjm\" (UniqueName: \"kubernetes.io/projected/50840f14-a207-4145-b855-aea24f6fa722-kube-api-access-5vcjm\") pod \"50840f14-a207-4145-b855-aea24f6fa722\" (UID: \"50840f14-a207-4145-b855-aea24f6fa722\") " Oct 30 20:47:27.186216 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.186094 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50840f14-a207-4145-b855-aea24f6fa722-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50840f14-a207-4145-b855-aea24f6fa722-plugins-config-volume\") pod \"50840f14-a207-4145-b855-aea24f6fa722\" (UID: \"50840f14-a207-4145-b855-aea24f6fa722\") " Oct 30 20:47:27.186327 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.186268 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cb1d9de-4456-4aee-9968-27eaa4be7479-kube-api-access-hd89z\" (UniqueName: \"kubernetes.io/projected/3cb1d9de-4456-4aee-9968-27eaa4be7479-kube-api-access-hd89z\") pod \"3cb1d9de-4456-4aee-9968-27eaa4be7479\" (UID: \"3cb1d9de-4456-4aee-9968-27eaa4be7479\") " Oct 30 20:47:27.186426 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.186409 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50840f14-a207-4145-b855-aea24f6fa722-plugins-config-volume" pod "50840f14-a207-4145-b855-aea24f6fa722" (UID: "50840f14-a207-4145-b855-aea24f6fa722"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:47:27.186483 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.186443 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb1d9de-4456-4aee-9968-27eaa4be7479-plugins-config-volume" pod "3cb1d9de-4456-4aee-9968-27eaa4be7479" (UID: "3cb1d9de-4456-4aee-9968-27eaa4be7479"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:47:27.186659 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.186641 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb1d9de-4456-4aee-9968-27eaa4be7479-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:27.186716 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.186666 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50840f14-a207-4145-b855-aea24f6fa722-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:27.188032 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.188012 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50840f14-a207-4145-b855-aea24f6fa722-kube-api-access-5vcjm" pod "50840f14-a207-4145-b855-aea24f6fa722" (UID: "50840f14-a207-4145-b855-aea24f6fa722"). InnerVolumeSpecName "kube-api-access-5vcjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:27.188102 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.188083 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb1d9de-4456-4aee-9968-27eaa4be7479-kube-api-access-hd89z" pod "3cb1d9de-4456-4aee-9968-27eaa4be7479" (UID: "3cb1d9de-4456-4aee-9968-27eaa4be7479"). InnerVolumeSpecName "kube-api-access-hd89z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:27.287959 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.287885 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vcjm\" (UniqueName: \"kubernetes.io/projected/50840f14-a207-4145-b855-aea24f6fa722-kube-api-access-5vcjm\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:27.287959 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.287909 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd89z\" (UniqueName: \"kubernetes.io/projected/3cb1d9de-4456-4aee-9968-27eaa4be7479-kube-api-access-hd89z\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:27.330347 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.330318 4848 generic.go:356] "Generic (PLEG): container finished" podID="3cb1d9de-4456-4aee-9968-27eaa4be7479" containerID="8571aaba7a867c7304724b8c1cab66cbc1e00741e3979e860e91606218190d76" exitCode=0 Oct 30 20:47:27.330496 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.330366 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6" Oct 30 20:47:27.330496 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.330407 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6" event={"ID":"3cb1d9de-4456-4aee-9968-27eaa4be7479","Type":"ContainerDied","Data":"8571aaba7a867c7304724b8c1cab66cbc1e00741e3979e860e91606218190d76"} Oct 30 20:47:27.330496 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.330465 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6" event={"ID":"3cb1d9de-4456-4aee-9968-27eaa4be7479","Type":"ContainerDied","Data":"d827594dac25c2536941ad74fdbc46eb817d453d1cbad19995c1521731bd9a97"} Oct 30 20:47:27.330644 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.330501 4848 scope.go:122] "RemoveContainer" containerID="8571aaba7a867c7304724b8c1cab66cbc1e00741e3979e860e91606218190d76" Oct 30 20:47:27.331786 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.331736 4848 generic.go:356] "Generic (PLEG): container finished" podID="50840f14-a207-4145-b855-aea24f6fa722" containerID="90ff0b1c69e0c9acb924e368b314aca095d2e424b09d78a0b87110c5400daa2a" exitCode=0 Oct 30 20:47:27.331905 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.331797 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml" Oct 30 20:47:27.331905 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.331810 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml" event={"ID":"50840f14-a207-4145-b855-aea24f6fa722","Type":"ContainerDied","Data":"90ff0b1c69e0c9acb924e368b314aca095d2e424b09d78a0b87110c5400daa2a"} Oct 30 20:47:27.331905 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.331861 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml" event={"ID":"50840f14-a207-4145-b855-aea24f6fa722","Type":"ContainerDied","Data":"f8aac83b9edb7f07cfbf8ce5fced5929a1baccb399c20cfded0ac61d449236ed"} Oct 30 20:47:27.339851 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.339756 4848 scope.go:122] "RemoveContainer" containerID="8571aaba7a867c7304724b8c1cab66cbc1e00741e3979e860e91606218190d76" Oct 30 20:47:27.340088 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:47:27.340061 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8571aaba7a867c7304724b8c1cab66cbc1e00741e3979e860e91606218190d76\": container with ID starting with 8571aaba7a867c7304724b8c1cab66cbc1e00741e3979e860e91606218190d76 not found: ID does not exist" containerID="8571aaba7a867c7304724b8c1cab66cbc1e00741e3979e860e91606218190d76" Oct 30 20:47:27.340193 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.340095 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8571aaba7a867c7304724b8c1cab66cbc1e00741e3979e860e91606218190d76"} err="failed to get container status \"8571aaba7a867c7304724b8c1cab66cbc1e00741e3979e860e91606218190d76\": rpc error: code = NotFound desc = could not find container \"8571aaba7a867c7304724b8c1cab66cbc1e00741e3979e860e91606218190d76\": container with ID starting with 8571aaba7a867c7304724b8c1cab66cbc1e00741e3979e860e91606218190d76 not found: ID does not exist" Oct 30 20:47:27.340193 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.340118 4848 scope.go:122] "RemoveContainer" containerID="90ff0b1c69e0c9acb924e368b314aca095d2e424b09d78a0b87110c5400daa2a" Oct 30 20:47:27.350537 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.350516 4848 scope.go:122] "RemoveContainer" containerID="90ff0b1c69e0c9acb924e368b314aca095d2e424b09d78a0b87110c5400daa2a" Oct 30 20:47:27.350996 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:47:27.350972 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90ff0b1c69e0c9acb924e368b314aca095d2e424b09d78a0b87110c5400daa2a\": container with ID starting with 90ff0b1c69e0c9acb924e368b314aca095d2e424b09d78a0b87110c5400daa2a not found: ID does not exist" containerID="90ff0b1c69e0c9acb924e368b314aca095d2e424b09d78a0b87110c5400daa2a" Oct 30 20:47:27.351091 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.351001 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90ff0b1c69e0c9acb924e368b314aca095d2e424b09d78a0b87110c5400daa2a"} err="failed to get container status \"90ff0b1c69e0c9acb924e368b314aca095d2e424b09d78a0b87110c5400daa2a\": rpc error: code = NotFound desc = could not find container \"90ff0b1c69e0c9acb924e368b314aca095d2e424b09d78a0b87110c5400daa2a\": container with ID starting with 90ff0b1c69e0c9acb924e368b314aca095d2e424b09d78a0b87110c5400daa2a not found: ID does not exist" Oct 30 20:47:27.364647 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.364621 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml"] Oct 30 20:47:27.367117 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.367093 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b612c7f2-8486646d87-sjvml"] Oct 30 20:47:27.377094 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.377070 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6"] Oct 30 20:47:27.383654 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.383633 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b612c7f2-75746f7d9-xh6h6"] Oct 30 20:47:27.787137 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.787104 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq"] Oct 30 20:47:27.787382 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:27.787354 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" podUID="fd9df289-7dc4-4c00-9b75-d57dd17656f4" containerName="istio-proxy" containerID="cri-o://fe0ea8cb3666cc5ace7a83c24d313efc9d19cae60ffc5cdaeef6669f454f1265" gracePeriod=30 Oct 30 20:47:28.931150 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:28.931117 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cb1d9de-4456-4aee-9968-27eaa4be7479" path="/var/lib/kubelet/pods/3cb1d9de-4456-4aee-9968-27eaa4be7479/volumes" Oct 30 20:47:28.931532 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:28.931485 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50840f14-a207-4145-b855-aea24f6fa722" path="/var/lib/kubelet/pods/50840f14-a207-4145-b855-aea24f6fa722/volumes" Oct 30 20:47:29.098677 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.098647 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf"] Oct 30 20:47:29.099120 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.099105 4848 cpu_manager.go:414] "containerMap: removing container" podUID="9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e" containerName="maas-api" Oct 30 20:47:29.099171 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.099121 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e" containerName="maas-api" Oct 30 20:47:29.099171 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.099142 4848 cpu_manager.go:414] "containerMap: removing container" podUID="3cb1d9de-4456-4aee-9968-27eaa4be7479" containerName="payload-processing" Oct 30 20:47:29.099171 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.099147 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cb1d9de-4456-4aee-9968-27eaa4be7479" containerName="payload-processing" Oct 30 20:47:29.099171 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.099155 4848 cpu_manager.go:414] "containerMap: removing container" podUID="50840f14-a207-4145-b855-aea24f6fa722" containerName="payload-pre-processing" Oct 30 20:47:29.099171 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.099160 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50840f14-a207-4145-b855-aea24f6fa722" containerName="payload-pre-processing" Oct 30 20:47:29.099320 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.099227 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c875fb6-8fab-46ee-8d8c-5adb3d8b5e1e" containerName="maas-api" Oct 30 20:47:29.099320 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.099238 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cb1d9de-4456-4aee-9968-27eaa4be7479" containerName="payload-processing" Oct 30 20:47:29.099320 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.099244 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="50840f14-a207-4145-b855-aea24f6fa722" containerName="payload-pre-processing" Oct 30 20:47:29.104170 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.104151 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.111179 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.111158 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-53aa47-openshift-default-dockercfg-flpcg\"" Oct 30 20:47:29.121966 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.121942 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf"] Oct 30 20:47:29.208343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.208261 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-data\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.208343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.208326 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-workload-certs\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.208602 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.208476 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57add2c-5ccf-47eb-8adb-d5852289b111-istiod-ca-cert\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.208602 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.208543 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-envoy\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.208716 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.208608 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-podinfo\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.208716 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.208650 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-token\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.208881 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.208783 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-workload-socket\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.208967 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.208893 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2528m\" (UniqueName: \"kubernetes.io/projected/e57add2c-5ccf-47eb-8adb-d5852289b111-kube-api-access-2528m\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.208967 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.208923 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-credential-socket\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.309851 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.309820 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2528m\" (UniqueName: \"kubernetes.io/projected/e57add2c-5ccf-47eb-8adb-d5852289b111-kube-api-access-2528m\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.310026 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.309859 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-credential-socket\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.310026 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.309892 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-data\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.310026 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.309929 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-workload-certs\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.310026 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.309984 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57add2c-5ccf-47eb-8adb-d5852289b111-istiod-ca-cert\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.310026 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.310024 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-envoy\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.310313 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.310047 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-podinfo\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.310313 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.310071 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-token\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.310313 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.310139 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-workload-socket\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.310313 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.310293 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-credential-socket\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.310538 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.310362 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-data\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.310538 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.310467 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-workload-certs\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.310538 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.310528 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-workload-socket\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.310801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.310762 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57add2c-5ccf-47eb-8adb-d5852289b111-istiod-ca-cert\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.312264 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.312244 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-envoy\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.312490 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.312472 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-podinfo\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.325615 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.325582 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-token\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.325834 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.325804 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2528m\" (UniqueName: \"kubernetes.io/projected/e57add2c-5ccf-47eb-8adb-d5852289b111-kube-api-access-2528m\") pod \"e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.418125 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.418093 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:29.607834 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.607804 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq"] Oct 30 20:47:29.608021 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.607997 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" podUID="59c21318-1027-43ea-8f08-7f67b4d097ed" containerName="maas-api" containerID="cri-o://3e97b85d4afc89d7ff5d97395e663d9cb8bcdca8a551bcc13dceeac00f55e2a1" gracePeriod=30 Oct 30 20:47:29.700205 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.700140 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf"] Oct 30 20:47:29.702603 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:47:29.702569 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57add2c_5ccf_47eb_8adb_d5852289b111.slice/crio-39403cc954c0004505ab47741e5bfb9f04db9ed09606dd8b0bd0458f278f6039 WatchSource:0}: Error finding container 39403cc954c0004505ab47741e5bfb9f04db9ed09606dd8b0bd0458f278f6039: Status 404 returned error can't find the container with id 39403cc954c0004505ab47741e5bfb9f04db9ed09606dd8b0bd0458f278f6039 Oct 30 20:47:29.714034 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.713995 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:29.714109 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.714084 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:29.714164 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.714147 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:29.874318 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.873360 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" Oct 30 20:47:29.916600 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.916570 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c21318-1027-43ea-8f08-7f67b4d097ed-kube-api-access-29ccr\" (UniqueName: \"kubernetes.io/projected/59c21318-1027-43ea-8f08-7f67b4d097ed-kube-api-access-29ccr\") pod \"59c21318-1027-43ea-8f08-7f67b4d097ed\" (UID: \"59c21318-1027-43ea-8f08-7f67b4d097ed\") " Oct 30 20:47:29.916814 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.916682 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59c21318-1027-43ea-8f08-7f67b4d097ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/59c21318-1027-43ea-8f08-7f67b4d097ed-metrics-certs\") pod \"59c21318-1027-43ea-8f08-7f67b4d097ed\" (UID: \"59c21318-1027-43ea-8f08-7f67b4d097ed\") " Oct 30 20:47:29.916903 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.916837 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c21318-1027-43ea-8f08-7f67b4d097ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59c21318-1027-43ea-8f08-7f67b4d097ed-openshift-service-ca\") pod \"59c21318-1027-43ea-8f08-7f67b4d097ed\" (UID: \"59c21318-1027-43ea-8f08-7f67b4d097ed\") " Oct 30 20:47:29.916903 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.916881 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59c21318-1027-43ea-8f08-7f67b4d097ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59c21318-1027-43ea-8f08-7f67b4d097ed-maas-api-tls\") pod \"59c21318-1027-43ea-8f08-7f67b4d097ed\" (UID: \"59c21318-1027-43ea-8f08-7f67b4d097ed\") " Oct 30 20:47:29.917283 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.917237 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c21318-1027-43ea-8f08-7f67b4d097ed-openshift-service-ca" pod "59c21318-1027-43ea-8f08-7f67b4d097ed" (UID: "59c21318-1027-43ea-8f08-7f67b4d097ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:47:29.918866 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.918843 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c21318-1027-43ea-8f08-7f67b4d097ed-metrics-certs" pod "59c21318-1027-43ea-8f08-7f67b4d097ed" (UID: "59c21318-1027-43ea-8f08-7f67b4d097ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:47:29.919013 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.918998 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c21318-1027-43ea-8f08-7f67b4d097ed-kube-api-access-29ccr" pod "59c21318-1027-43ea-8f08-7f67b4d097ed" (UID: "59c21318-1027-43ea-8f08-7f67b4d097ed"). InnerVolumeSpecName "kube-api-access-29ccr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:29.919320 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:29.919299 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c21318-1027-43ea-8f08-7f67b4d097ed-maas-api-tls" pod "59c21318-1027-43ea-8f08-7f67b4d097ed" (UID: "59c21318-1027-43ea-8f08-7f67b4d097ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:47:30.025719 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.019340 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29ccr\" (UniqueName: \"kubernetes.io/projected/59c21318-1027-43ea-8f08-7f67b4d097ed-kube-api-access-29ccr\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:30.025719 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.019377 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59c21318-1027-43ea-8f08-7f67b4d097ed-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:30.025719 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.019399 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59c21318-1027-43ea-8f08-7f67b4d097ed-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:30.025719 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.019416 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59c21318-1027-43ea-8f08-7f67b4d097ed-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:30.131784 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.131744 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9"] Oct 30 20:47:30.131996 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.131974 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" podUID="2a2a74d1-f1a1-4434-bc2f-966fb5123180" containerName="istio-proxy" containerID="cri-o://9cc32722a334548cd9111c857749b19b60c3787016a7f936a858115b7722950b" gracePeriod=30 Oct 30 20:47:30.351591 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.351558 4848 generic.go:356] "Generic (PLEG): container finished" podID="59c21318-1027-43ea-8f08-7f67b4d097ed" containerID="3e97b85d4afc89d7ff5d97395e663d9cb8bcdca8a551bcc13dceeac00f55e2a1" exitCode=0 Oct 30 20:47:30.351802 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.351618 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" Oct 30 20:47:30.351802 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.351645 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" event={"ID":"59c21318-1027-43ea-8f08-7f67b4d097ed","Type":"ContainerDied","Data":"3e97b85d4afc89d7ff5d97395e663d9cb8bcdca8a551bcc13dceeac00f55e2a1"} Oct 30 20:47:30.351802 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.351706 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq" event={"ID":"59c21318-1027-43ea-8f08-7f67b4d097ed","Type":"ContainerDied","Data":"12216d8ccb12f5233638dfd26805db320728ffb92470d10f9c6a8a1a322c5aef"} Oct 30 20:47:30.351802 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.351741 4848 scope.go:122] "RemoveContainer" containerID="3e97b85d4afc89d7ff5d97395e663d9cb8bcdca8a551bcc13dceeac00f55e2a1" Oct 30 20:47:30.353932 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.353908 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" event={"ID":"e57add2c-5ccf-47eb-8adb-d5852289b111","Type":"ContainerStarted","Data":"8e74ab0c9e3f60b54bdd7640b98965b377814024591e577b3ef4af4a82d75b86"} Oct 30 20:47:30.354039 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.353958 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" event={"ID":"e57add2c-5ccf-47eb-8adb-d5852289b111","Type":"ContainerStarted","Data":"39403cc954c0004505ab47741e5bfb9f04db9ed09606dd8b0bd0458f278f6039"} Oct 30 20:47:30.361345 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.361328 4848 scope.go:122] "RemoveContainer" containerID="3e97b85d4afc89d7ff5d97395e663d9cb8bcdca8a551bcc13dceeac00f55e2a1" Oct 30 20:47:30.361612 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:47:30.361594 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e97b85d4afc89d7ff5d97395e663d9cb8bcdca8a551bcc13dceeac00f55e2a1\": container with ID starting with 3e97b85d4afc89d7ff5d97395e663d9cb8bcdca8a551bcc13dceeac00f55e2a1 not found: ID does not exist" containerID="3e97b85d4afc89d7ff5d97395e663d9cb8bcdca8a551bcc13dceeac00f55e2a1" Oct 30 20:47:30.361678 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.361624 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e97b85d4afc89d7ff5d97395e663d9cb8bcdca8a551bcc13dceeac00f55e2a1"} err="failed to get container status \"3e97b85d4afc89d7ff5d97395e663d9cb8bcdca8a551bcc13dceeac00f55e2a1\": rpc error: code = NotFound desc = could not find container \"3e97b85d4afc89d7ff5d97395e663d9cb8bcdca8a551bcc13dceeac00f55e2a1\": container with ID starting with 3e97b85d4afc89d7ff5d97395e663d9cb8bcdca8a551bcc13dceeac00f55e2a1 not found: ID does not exist" Oct 30 20:47:30.381399 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.381358 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" podStartSLOduration=1.3813462429999999 podStartE2EDuration="1.381346243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:30.379188815 +0000 UTC m=+555.559085321" watchObservedRunningTime="2026-10-30 20:47:30.381346243 +0000 UTC m=+555.561242749" Oct 30 20:47:30.397855 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.397833 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq"] Oct 30 20:47:30.409327 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.409308 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-8f46c771-76c658df6b-n8tdq"] Oct 30 20:47:30.418496 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.418472 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:30.423696 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.423672 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:30.940176 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:30.940142 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59c21318-1027-43ea-8f08-7f67b4d097ed" path="/var/lib/kubelet/pods/59c21318-1027-43ea-8f08-7f67b4d097ed/volumes" Oct 30 20:47:31.347479 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.347396 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84"] Oct 30 20:47:31.348027 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.348007 4848 cpu_manager.go:414] "containerMap: removing container" podUID="59c21318-1027-43ea-8f08-7f67b4d097ed" containerName="maas-api" Oct 30 20:47:31.348114 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.348028 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c21318-1027-43ea-8f08-7f67b4d097ed" containerName="maas-api" Oct 30 20:47:31.348183 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.348169 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c21318-1027-43ea-8f08-7f67b4d097ed" containerName="maas-api" Oct 30 20:47:31.352653 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.352631 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.359144 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.359106 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-f9a819-openshift-default-dockercfg-925js\"" Oct 30 20:47:31.362133 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.362071 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:31.363471 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.363445 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:31.365489 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.365464 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84"] Oct 30 20:47:31.438202 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.438140 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.438202 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.438199 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.439505 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.438243 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.439505 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.438272 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g22g4\" (UniqueName: \"kubernetes.io/projected/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-kube-api-access-g22g4\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.439505 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.438529 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.439505 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.438589 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.439505 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.438628 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.439505 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.438656 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.439505 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.438686 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.542041 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.539628 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.542041 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.539681 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.542041 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.539812 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.542041 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.539849 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.542041 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.539887 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.542041 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.539918 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g22g4\" (UniqueName: \"kubernetes.io/projected/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-kube-api-access-g22g4\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.542041 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.540024 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.542041 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.540065 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.542041 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.540093 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.542041 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.540459 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.542041 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.540703 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.542041 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.541277 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.542041 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.541746 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.542041 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.541988 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.543136 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.543105 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.543327 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.543306 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.564570 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.564503 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.565430 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.565376 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g22g4\" (UniqueName: \"kubernetes.io/projected/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-kube-api-access-g22g4\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.668180 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.668093 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:31.987183 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.987152 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84"] Oct 30 20:47:31.988959 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:47:31.988930 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8056f3a1_1b99_4dcd_8d2d_0a1a386cc1c4.slice/crio-477703af9b146492f0a38d19489b96b07d0702e0803c4748efa615677c2eb603 WatchSource:0}: Error finding container 477703af9b146492f0a38d19489b96b07d0702e0803c4748efa615677c2eb603: Status 404 returned error can't find the container with id 477703af9b146492f0a38d19489b96b07d0702e0803c4748efa615677c2eb603 Oct 30 20:47:31.991292 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.991258 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:31.991370 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.991329 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:31.991410 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:31.991392 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:32.367972 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:32.367883 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" event={"ID":"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4","Type":"ContainerStarted","Data":"fa2a27fc7a0dc4cf57ca23ba474f5227396d6e7d6ad364b24dd0edbbc205e11f"} Oct 30 20:47:32.367972 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:32.367946 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" event={"ID":"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4","Type":"ContainerStarted","Data":"477703af9b146492f0a38d19489b96b07d0702e0803c4748efa615677c2eb603"} Oct 30 20:47:32.396503 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:32.396454 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" podStartSLOduration=1.396439711 podStartE2EDuration="1.396439711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:32.393399179 +0000 UTC m=+557.573295692" watchObservedRunningTime="2026-10-30 20:47:32.396439711 +0000 UTC m=+557.576336217" Oct 30 20:47:32.654500 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:32.654423 4848 patch_prober.go:29] interesting pod/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:47:32.654500 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:32.654478 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" podUID="fd9df289-7dc4-4c00-9b75-d57dd17656f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:47:32.668184 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:32.668145 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:33.056476 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.056452 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:47:33.156263 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.156234 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-podinfo\") pod \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " Oct 30 20:47:33.156439 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.156271 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-workload-certs\") pod \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " Oct 30 20:47:33.156439 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.156314 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-token\" (UniqueName: \"kubernetes.io/projected/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-token\") pod \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " Oct 30 20:47:33.156439 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.156356 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istiod-ca-cert\") pod \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " Oct 30 20:47:33.156439 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.156387 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-data\") pod \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " Oct 30 20:47:33.156439 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.156416 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-envoy\") pod \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " Oct 30 20:47:33.156439 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.156439 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-credential-socket\") pod \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " Oct 30 20:47:33.156759 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.156490 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-workload-socket\") pod \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " Oct 30 20:47:33.156759 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.156528 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd9df289-7dc4-4c00-9b75-d57dd17656f4-kube-api-access-dshzp\" (UniqueName: \"kubernetes.io/projected/fd9df289-7dc4-4c00-9b75-d57dd17656f4-kube-api-access-dshzp\") pod \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\" (UID: \"fd9df289-7dc4-4c00-9b75-d57dd17656f4\") " Oct 30 20:47:33.156759 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.156668 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-workload-certs" pod "fd9df289-7dc4-4c00-9b75-d57dd17656f4" (UID: "fd9df289-7dc4-4c00-9b75-d57dd17656f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:47:33.156759 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.156685 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-data" pod "fd9df289-7dc4-4c00-9b75-d57dd17656f4" (UID: "fd9df289-7dc4-4c00-9b75-d57dd17656f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:47:33.156759 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.156751 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istiod-ca-cert" pod "fd9df289-7dc4-4c00-9b75-d57dd17656f4" (UID: "fd9df289-7dc4-4c00-9b75-d57dd17656f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:47:33.157124 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.156941 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-credential-socket" pod "fd9df289-7dc4-4c00-9b75-d57dd17656f4" (UID: "fd9df289-7dc4-4c00-9b75-d57dd17656f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:47:33.157124 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.156984 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:33.157124 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.157006 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:33.157124 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.157019 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:33.157124 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.157035 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-workload-socket" pod "fd9df289-7dc4-4c00-9b75-d57dd17656f4" (UID: "fd9df289-7dc4-4c00-9b75-d57dd17656f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:47:33.158559 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.158495 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-envoy" pod "fd9df289-7dc4-4c00-9b75-d57dd17656f4" (UID: "fd9df289-7dc4-4c00-9b75-d57dd17656f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:47:33.158673 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.158554 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-podinfo" pod "fd9df289-7dc4-4c00-9b75-d57dd17656f4" (UID: "fd9df289-7dc4-4c00-9b75-d57dd17656f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:47:33.158673 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.158616 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-token" pod "fd9df289-7dc4-4c00-9b75-d57dd17656f4" (UID: "fd9df289-7dc4-4c00-9b75-d57dd17656f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:33.158673 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.158641 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9df289-7dc4-4c00-9b75-d57dd17656f4-kube-api-access-dshzp" pod "fd9df289-7dc4-4c00-9b75-d57dd17656f4" (UID: "fd9df289-7dc4-4c00-9b75-d57dd17656f4"). InnerVolumeSpecName "kube-api-access-dshzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:33.258132 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.258088 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:33.258132 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.258116 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:33.258132 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.258126 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:33.258132 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.258135 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:33.258132 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.258144 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9df289-7dc4-4c00-9b75-d57dd17656f4-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:33.258525 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.258153 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dshzp\" (UniqueName: \"kubernetes.io/projected/fd9df289-7dc4-4c00-9b75-d57dd17656f4-kube-api-access-dshzp\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:33.375038 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.375001 4848 generic.go:356] "Generic (PLEG): container finished" podID="fd9df289-7dc4-4c00-9b75-d57dd17656f4" containerID="fe0ea8cb3666cc5ace7a83c24d313efc9d19cae60ffc5cdaeef6669f454f1265" exitCode=0 Oct 30 20:47:33.375504 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.375052 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" Oct 30 20:47:33.375504 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.375083 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" event={"ID":"fd9df289-7dc4-4c00-9b75-d57dd17656f4","Type":"ContainerDied","Data":"fe0ea8cb3666cc5ace7a83c24d313efc9d19cae60ffc5cdaeef6669f454f1265"} Oct 30 20:47:33.375504 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.375134 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq" event={"ID":"fd9df289-7dc4-4c00-9b75-d57dd17656f4","Type":"ContainerDied","Data":"dc06c2aadd6142885d68c29fb0aba6b46bc011827d3bb59968cfdda5a46c4a2e"} Oct 30 20:47:33.375504 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.375155 4848 scope.go:122] "RemoveContainer" containerID="fe0ea8cb3666cc5ace7a83c24d313efc9d19cae60ffc5cdaeef6669f454f1265" Oct 30 20:47:33.385036 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.385018 4848 scope.go:122] "RemoveContainer" containerID="98a23860368255afe482d86497b28db63aec70939576c01430f3948ecf539e73" Oct 30 20:47:33.393123 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.393106 4848 scope.go:122] "RemoveContainer" containerID="fe0ea8cb3666cc5ace7a83c24d313efc9d19cae60ffc5cdaeef6669f454f1265" Oct 30 20:47:33.396809 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:47:33.396751 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe0ea8cb3666cc5ace7a83c24d313efc9d19cae60ffc5cdaeef6669f454f1265\": container with ID starting with fe0ea8cb3666cc5ace7a83c24d313efc9d19cae60ffc5cdaeef6669f454f1265 not found: ID does not exist" containerID="fe0ea8cb3666cc5ace7a83c24d313efc9d19cae60ffc5cdaeef6669f454f1265" Oct 30 20:47:33.396899 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.396820 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe0ea8cb3666cc5ace7a83c24d313efc9d19cae60ffc5cdaeef6669f454f1265"} err="failed to get container status \"fe0ea8cb3666cc5ace7a83c24d313efc9d19cae60ffc5cdaeef6669f454f1265\": rpc error: code = NotFound desc = could not find container \"fe0ea8cb3666cc5ace7a83c24d313efc9d19cae60ffc5cdaeef6669f454f1265\": container with ID starting with fe0ea8cb3666cc5ace7a83c24d313efc9d19cae60ffc5cdaeef6669f454f1265 not found: ID does not exist" Oct 30 20:47:33.396899 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.396838 4848 scope.go:122] "RemoveContainer" containerID="98a23860368255afe482d86497b28db63aec70939576c01430f3948ecf539e73" Oct 30 20:47:33.397431 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:47:33.397409 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98a23860368255afe482d86497b28db63aec70939576c01430f3948ecf539e73\": container with ID starting with 98a23860368255afe482d86497b28db63aec70939576c01430f3948ecf539e73 not found: ID does not exist" containerID="98a23860368255afe482d86497b28db63aec70939576c01430f3948ecf539e73" Oct 30 20:47:33.397502 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.397441 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98a23860368255afe482d86497b28db63aec70939576c01430f3948ecf539e73"} err="failed to get container status \"98a23860368255afe482d86497b28db63aec70939576c01430f3948ecf539e73\": rpc error: code = NotFound desc = could not find container \"98a23860368255afe482d86497b28db63aec70939576c01430f3948ecf539e73\": container with ID starting with 98a23860368255afe482d86497b28db63aec70939576c01430f3948ecf539e73 not found: ID does not exist" Oct 30 20:47:33.399440 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.399421 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq"] Oct 30 20:47:33.404424 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.404389 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b612c7f2-openshift-default-6fb7f7b587-nsgrq"] Oct 30 20:47:33.673903 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:33.673871 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:34.382001 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:34.381965 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:34.382908 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:34.382888 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:34.453826 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:34.453795 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5"] Oct 30 20:47:34.932620 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:34.932580 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd9df289-7dc4-4c00-9b75-d57dd17656f4" path="/var/lib/kubelet/pods/fd9df289-7dc4-4c00-9b75-d57dd17656f4/volumes" Oct 30 20:47:35.380101 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.380073 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:47:35.387329 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.387300 4848 generic.go:356] "Generic (PLEG): container finished" podID="2a2a74d1-f1a1-4434-bc2f-966fb5123180" containerID="9cc32722a334548cd9111c857749b19b60c3787016a7f936a858115b7722950b" exitCode=0 Oct 30 20:47:35.387676 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.387341 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" Oct 30 20:47:35.387676 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.387384 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" event={"ID":"2a2a74d1-f1a1-4434-bc2f-966fb5123180","Type":"ContainerDied","Data":"9cc32722a334548cd9111c857749b19b60c3787016a7f936a858115b7722950b"} Oct 30 20:47:35.387676 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.387434 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9" event={"ID":"2a2a74d1-f1a1-4434-bc2f-966fb5123180","Type":"ContainerDied","Data":"2f424d8dff7547baa085a9fc29c3cbe9dd57515af500aad19f113a0bcf01bba7"} Oct 30 20:47:35.387676 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.387458 4848 scope.go:122] "RemoveContainer" containerID="9cc32722a334548cd9111c857749b19b60c3787016a7f936a858115b7722950b" Oct 30 20:47:35.395872 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.395856 4848 scope.go:122] "RemoveContainer" containerID="0e16f4acf08a5799830c7ce55f9c97d6bed4bb555500535d3a2b156c5ea42e15" Oct 30 20:47:35.403466 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.403451 4848 scope.go:122] "RemoveContainer" containerID="9cc32722a334548cd9111c857749b19b60c3787016a7f936a858115b7722950b" Oct 30 20:47:35.403734 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:47:35.403715 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cc32722a334548cd9111c857749b19b60c3787016a7f936a858115b7722950b\": container with ID starting with 9cc32722a334548cd9111c857749b19b60c3787016a7f936a858115b7722950b not found: ID does not exist" containerID="9cc32722a334548cd9111c857749b19b60c3787016a7f936a858115b7722950b" Oct 30 20:47:35.403796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.403744 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc32722a334548cd9111c857749b19b60c3787016a7f936a858115b7722950b"} err="failed to get container status \"9cc32722a334548cd9111c857749b19b60c3787016a7f936a858115b7722950b\": rpc error: code = NotFound desc = could not find container \"9cc32722a334548cd9111c857749b19b60c3787016a7f936a858115b7722950b\": container with ID starting with 9cc32722a334548cd9111c857749b19b60c3787016a7f936a858115b7722950b not found: ID does not exist" Oct 30 20:47:35.403862 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.403762 4848 scope.go:122] "RemoveContainer" containerID="0e16f4acf08a5799830c7ce55f9c97d6bed4bb555500535d3a2b156c5ea42e15" Oct 30 20:47:35.404098 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:47:35.404082 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e16f4acf08a5799830c7ce55f9c97d6bed4bb555500535d3a2b156c5ea42e15\": container with ID starting with 0e16f4acf08a5799830c7ce55f9c97d6bed4bb555500535d3a2b156c5ea42e15 not found: ID does not exist" containerID="0e16f4acf08a5799830c7ce55f9c97d6bed4bb555500535d3a2b156c5ea42e15" Oct 30 20:47:35.404160 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.404105 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e16f4acf08a5799830c7ce55f9c97d6bed4bb555500535d3a2b156c5ea42e15"} err="failed to get container status \"0e16f4acf08a5799830c7ce55f9c97d6bed4bb555500535d3a2b156c5ea42e15\": rpc error: code = NotFound desc = could not find container \"0e16f4acf08a5799830c7ce55f9c97d6bed4bb555500535d3a2b156c5ea42e15\": container with ID starting with 0e16f4acf08a5799830c7ce55f9c97d6bed4bb555500535d3a2b156c5ea42e15 not found: ID does not exist" Oct 30 20:47:35.485335 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.485305 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-workload-certs\") pod \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " Oct 30 20:47:35.485335 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.485337 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-workload-socket\") pod \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " Oct 30 20:47:35.485542 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.485356 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-credential-socket\") pod \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " Oct 30 20:47:35.485542 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.485379 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-podinfo\") pod \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " Oct 30 20:47:35.485542 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.485429 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istiod-ca-cert\") pod \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " Oct 30 20:47:35.485542 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.485495 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a2a74d1-f1a1-4434-bc2f-966fb5123180-kube-api-access-cgsdb\" (UniqueName: \"kubernetes.io/projected/2a2a74d1-f1a1-4434-bc2f-966fb5123180-kube-api-access-cgsdb\") pod \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " Oct 30 20:47:35.485542 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.485521 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-token\" (UniqueName: \"kubernetes.io/projected/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-token\") pod \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " Oct 30 20:47:35.485542 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.485539 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-envoy\") pod \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " Oct 30 20:47:35.485863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.485592 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-data\") pod \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\" (UID: \"2a2a74d1-f1a1-4434-bc2f-966fb5123180\") " Oct 30 20:47:35.485863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.485610 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-workload-socket" pod "2a2a74d1-f1a1-4434-bc2f-966fb5123180" (UID: "2a2a74d1-f1a1-4434-bc2f-966fb5123180"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:47:35.485863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.485676 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-credential-socket" pod "2a2a74d1-f1a1-4434-bc2f-966fb5123180" (UID: "2a2a74d1-f1a1-4434-bc2f-966fb5123180"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:47:35.485863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.485696 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-workload-certs" pod "2a2a74d1-f1a1-4434-bc2f-966fb5123180" (UID: "2a2a74d1-f1a1-4434-bc2f-966fb5123180"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:47:35.486043 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.485907 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-data" pod "2a2a74d1-f1a1-4434-bc2f-966fb5123180" (UID: "2a2a74d1-f1a1-4434-bc2f-966fb5123180"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:47:35.486043 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.485965 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istiod-ca-cert" pod "2a2a74d1-f1a1-4434-bc2f-966fb5123180" (UID: "2a2a74d1-f1a1-4434-bc2f-966fb5123180"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:47:35.486139 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.486116 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:35.486139 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.486135 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:35.486241 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.486150 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:35.486241 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.486163 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:35.486241 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.486191 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:35.487942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.487913 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2a74d1-f1a1-4434-bc2f-966fb5123180-kube-api-access-cgsdb" pod "2a2a74d1-f1a1-4434-bc2f-966fb5123180" (UID: "2a2a74d1-f1a1-4434-bc2f-966fb5123180"). InnerVolumeSpecName "kube-api-access-cgsdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:35.487942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.487914 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-envoy" pod "2a2a74d1-f1a1-4434-bc2f-966fb5123180" (UID: "2a2a74d1-f1a1-4434-bc2f-966fb5123180"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:47:35.488118 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.488086 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-token" pod "2a2a74d1-f1a1-4434-bc2f-966fb5123180" (UID: "2a2a74d1-f1a1-4434-bc2f-966fb5123180"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:35.488172 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.488127 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-podinfo" pod "2a2a74d1-f1a1-4434-bc2f-966fb5123180" (UID: "2a2a74d1-f1a1-4434-bc2f-966fb5123180"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:47:35.587326 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.587261 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgsdb\" (UniqueName: \"kubernetes.io/projected/2a2a74d1-f1a1-4434-bc2f-966fb5123180-kube-api-access-cgsdb\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:35.587326 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.587286 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:35.587326 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.587301 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:35.587326 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.587312 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a2a74d1-f1a1-4434-bc2f-966fb5123180-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:35.720398 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.720358 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9"] Oct 30 20:47:35.726917 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:35.726891 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-8f46c771-gw-openshift-default-76467b7b9-z2mn9"] Oct 30 20:47:36.127709 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.106964 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx"] Oct 30 20:47:36.131345 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.131312 4848 cpu_manager.go:414] "containerMap: removing container" podUID="fd9df289-7dc4-4c00-9b75-d57dd17656f4" containerName="istio-proxy" Oct 30 20:47:36.131581 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.131562 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd9df289-7dc4-4c00-9b75-d57dd17656f4" containerName="istio-proxy" Oct 30 20:47:36.132593 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.131741 4848 cpu_manager.go:414] "containerMap: removing container" podUID="2a2a74d1-f1a1-4434-bc2f-966fb5123180" containerName="istio-proxy" Oct 30 20:47:36.132841 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.132710 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a2a74d1-f1a1-4434-bc2f-966fb5123180" containerName="istio-proxy" Oct 30 20:47:36.132989 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.132975 4848 cpu_manager.go:414] "containerMap: removing container" podUID="fd9df289-7dc4-4c00-9b75-d57dd17656f4" containerName="istio-proxy" Oct 30 20:47:36.133085 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.133073 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd9df289-7dc4-4c00-9b75-d57dd17656f4" containerName="istio-proxy" Oct 30 20:47:36.133307 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.133293 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a2a74d1-f1a1-4434-bc2f-966fb5123180" containerName="istio-proxy" Oct 30 20:47:36.133411 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.133400 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd9df289-7dc4-4c00-9b75-d57dd17656f4" containerName="istio-proxy" Oct 30 20:47:36.133488 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.133479 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd9df289-7dc4-4c00-9b75-d57dd17656f4" containerName="istio-proxy" Oct 30 20:47:36.139071 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.139039 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx"] Oct 30 20:47:36.139205 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.139166 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" Oct 30 20:47:36.150352 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.149791 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-53aa47-serving-cert\"" Oct 30 20:47:36.194811 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.194757 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f7e1ecd-c614-4687-96e3-631e5716cf83-maas-api-tls\") pod \"maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx\" (UID: \"9f7e1ecd-c614-4687-96e3-631e5716cf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" Oct 30 20:47:36.194964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.194892 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f7e1ecd-c614-4687-96e3-631e5716cf83-metrics-certs\") pod \"maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx\" (UID: \"9f7e1ecd-c614-4687-96e3-631e5716cf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" Oct 30 20:47:36.194964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.194918 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f7e1ecd-c614-4687-96e3-631e5716cf83-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx\" (UID: \"9f7e1ecd-c614-4687-96e3-631e5716cf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" Oct 30 20:47:36.195042 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.195028 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl489\" (UniqueName: \"kubernetes.io/projected/9f7e1ecd-c614-4687-96e3-631e5716cf83-kube-api-access-dl489\") pod \"maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx\" (UID: \"9f7e1ecd-c614-4687-96e3-631e5716cf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" Oct 30 20:47:36.296524 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.296487 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl489\" (UniqueName: \"kubernetes.io/projected/9f7e1ecd-c614-4687-96e3-631e5716cf83-kube-api-access-dl489\") pod \"maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx\" (UID: \"9f7e1ecd-c614-4687-96e3-631e5716cf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" Oct 30 20:47:36.296718 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.296540 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f7e1ecd-c614-4687-96e3-631e5716cf83-maas-api-tls\") pod \"maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx\" (UID: \"9f7e1ecd-c614-4687-96e3-631e5716cf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" Oct 30 20:47:36.296718 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.296583 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f7e1ecd-c614-4687-96e3-631e5716cf83-metrics-certs\") pod \"maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx\" (UID: \"9f7e1ecd-c614-4687-96e3-631e5716cf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" Oct 30 20:47:36.296718 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.296599 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f7e1ecd-c614-4687-96e3-631e5716cf83-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx\" (UID: \"9f7e1ecd-c614-4687-96e3-631e5716cf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" Oct 30 20:47:36.297331 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.297307 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f7e1ecd-c614-4687-96e3-631e5716cf83-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx\" (UID: \"9f7e1ecd-c614-4687-96e3-631e5716cf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" Oct 30 20:47:36.298975 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.298950 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f7e1ecd-c614-4687-96e3-631e5716cf83-metrics-certs\") pod \"maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx\" (UID: \"9f7e1ecd-c614-4687-96e3-631e5716cf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" Oct 30 20:47:36.299096 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.299044 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f7e1ecd-c614-4687-96e3-631e5716cf83-maas-api-tls\") pod \"maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx\" (UID: \"9f7e1ecd-c614-4687-96e3-631e5716cf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" Oct 30 20:47:36.309338 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.309315 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl489\" (UniqueName: \"kubernetes.io/projected/9f7e1ecd-c614-4687-96e3-631e5716cf83-kube-api-access-dl489\") pod \"maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx\" (UID: \"9f7e1ecd-c614-4687-96e3-631e5716cf83\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" Oct 30 20:47:36.454760 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.454726 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" Oct 30 20:47:36.707391 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.707278 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr"] Oct 30 20:47:36.708044 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.708020 4848 cpu_manager.go:414] "containerMap: removing container" podUID="2a2a74d1-f1a1-4434-bc2f-966fb5123180" containerName="istio-proxy" Oct 30 20:47:36.708044 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.708042 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a2a74d1-f1a1-4434-bc2f-966fb5123180" containerName="istio-proxy" Oct 30 20:47:36.708199 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.708153 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a2a74d1-f1a1-4434-bc2f-966fb5123180" containerName="istio-proxy" Oct 30 20:47:36.712220 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.712201 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr" Oct 30 20:47:36.725418 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.725389 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr"] Oct 30 20:47:36.729263 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.729222 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-f9a819\"" Oct 30 20:47:36.729429 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.729322 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-f9a819-dockercfg-2pwj2\"" Oct 30 20:47:36.743191 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.743166 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx"] Oct 30 20:47:36.744278 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:47:36.744253 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f7e1ecd_c614_4687_96e3_631e5716cf83.slice/crio-fe56079c3e08a4c67b8443ee9e66f6f9330da33a7c897efb7c810b33cdeec081 WatchSource:0}: Error finding container fe56079c3e08a4c67b8443ee9e66f6f9330da33a7c897efb7c810b33cdeec081: Status 404 returned error can't find the container with id fe56079c3e08a4c67b8443ee9e66f6f9330da33a7c897efb7c810b33cdeec081 Oct 30 20:47:36.753165 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.753143 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d"] Oct 30 20:47:36.756792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.756761 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d" Oct 30 20:47:36.765341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.765314 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d"] Oct 30 20:47:36.802099 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.802070 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6867e706-6187-4821-ae3c-141f443718a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr\" (UID: \"6867e706-6187-4821-ae3c-141f443718a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr" Oct 30 20:47:36.802232 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.802106 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8sbc\" (UniqueName: \"kubernetes.io/projected/6867e706-6187-4821-ae3c-141f443718a5-kube-api-access-s8sbc\") pod \"payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr\" (UID: \"6867e706-6187-4821-ae3c-141f443718a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr" Oct 30 20:47:36.802232 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.802195 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbk5j\" (UniqueName: \"kubernetes.io/projected/3d385f07-6dae-4ca6-9e02-eb587a7d6374-kube-api-access-mbk5j\") pod \"payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d\" (UID: \"3d385f07-6dae-4ca6-9e02-eb587a7d6374\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d" Oct 30 20:47:36.802319 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.802270 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d385f07-6dae-4ca6-9e02-eb587a7d6374-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d\" (UID: \"3d385f07-6dae-4ca6-9e02-eb587a7d6374\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d" Oct 30 20:47:36.881749 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.881718 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g"] Oct 30 20:47:36.903318 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.903290 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d385f07-6dae-4ca6-9e02-eb587a7d6374-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d\" (UID: \"3d385f07-6dae-4ca6-9e02-eb587a7d6374\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d" Oct 30 20:47:36.903488 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.903418 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6867e706-6187-4821-ae3c-141f443718a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr\" (UID: \"6867e706-6187-4821-ae3c-141f443718a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr" Oct 30 20:47:36.903488 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.903450 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8sbc\" (UniqueName: \"kubernetes.io/projected/6867e706-6187-4821-ae3c-141f443718a5-kube-api-access-s8sbc\") pod \"payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr\" (UID: \"6867e706-6187-4821-ae3c-141f443718a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr" Oct 30 20:47:36.903617 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.903496 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbk5j\" (UniqueName: \"kubernetes.io/projected/3d385f07-6dae-4ca6-9e02-eb587a7d6374-kube-api-access-mbk5j\") pod \"payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d\" (UID: \"3d385f07-6dae-4ca6-9e02-eb587a7d6374\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d" Oct 30 20:47:36.904279 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.904253 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d385f07-6dae-4ca6-9e02-eb587a7d6374-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d\" (UID: \"3d385f07-6dae-4ca6-9e02-eb587a7d6374\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d" Oct 30 20:47:36.905097 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.905068 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6867e706-6187-4821-ae3c-141f443718a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr\" (UID: \"6867e706-6187-4821-ae3c-141f443718a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr" Oct 30 20:47:36.920449 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.920424 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8sbc\" (UniqueName: \"kubernetes.io/projected/6867e706-6187-4821-ae3c-141f443718a5-kube-api-access-s8sbc\") pod \"payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr\" (UID: \"6867e706-6187-4821-ae3c-141f443718a5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr" Oct 30 20:47:36.920843 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.920820 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbk5j\" (UniqueName: \"kubernetes.io/projected/3d385f07-6dae-4ca6-9e02-eb587a7d6374-kube-api-access-mbk5j\") pod \"payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d\" (UID: \"3d385f07-6dae-4ca6-9e02-eb587a7d6374\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d" Oct 30 20:47:36.931387 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.931362 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a2a74d1-f1a1-4434-bc2f-966fb5123180" path="/var/lib/kubelet/pods/2a2a74d1-f1a1-4434-bc2f-966fb5123180/volumes" Oct 30 20:47:36.995705 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.995625 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld"] Oct 30 20:47:36.995982 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:36.995877 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" podUID="84be7e6d-711c-46f8-a188-0f8955560e5a" containerName="maas-api" containerID="cri-o://e42ce637b917a6f0381b8b509426cd35c0a55b0c67af3c771239a94efec64213" gracePeriod=30 Oct 30 20:47:37.028799 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.027961 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr" Oct 30 20:47:37.074584 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.074558 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d" Oct 30 20:47:37.180578 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.180549 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz"] Oct 30 20:47:37.180982 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.180877 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" podUID="39bb7b01-c262-4d9d-b919-74156bff3b41" containerName="maas-api" containerID="cri-o://fb00f5792fd61211a8cfeeb5b75ee3238b5bbe3e6815fd5650f16cd5fa0cbca6" gracePeriod=30 Oct 30 20:47:37.308583 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.308558 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr"] Oct 30 20:47:37.311805 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:47:37.311742 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6867e706_6187_4821_ae3c_141f443718a5.slice/crio-2d8478165477b897ce1ce25f429c8c022c09f5e9e372200f31d57a1db974ee67 WatchSource:0}: Error finding container 2d8478165477b897ce1ce25f429c8c022c09f5e9e372200f31d57a1db974ee67: Status 404 returned error can't find the container with id 2d8478165477b897ce1ce25f429c8c022c09f5e9e372200f31d57a1db974ee67 Oct 30 20:47:37.363942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.363780 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d"] Oct 30 20:47:37.402088 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.402018 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" event={"ID":"9f7e1ecd-c614-4687-96e3-631e5716cf83","Type":"ContainerStarted","Data":"fe56079c3e08a4c67b8443ee9e66f6f9330da33a7c897efb7c810b33cdeec081"} Oct 30 20:47:37.410224 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.410182 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr" event={"ID":"6867e706-6187-4821-ae3c-141f443718a5","Type":"ContainerStarted","Data":"2d8478165477b897ce1ce25f429c8c022c09f5e9e372200f31d57a1db974ee67"} Oct 30 20:47:37.468055 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.466251 4848 generic.go:356] "Generic (PLEG): container finished" podID="39bb7b01-c262-4d9d-b919-74156bff3b41" containerID="fb00f5792fd61211a8cfeeb5b75ee3238b5bbe3e6815fd5650f16cd5fa0cbca6" exitCode=0 Oct 30 20:47:37.468055 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.466413 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" event={"ID":"39bb7b01-c262-4d9d-b919-74156bff3b41","Type":"ContainerDied","Data":"fb00f5792fd61211a8cfeeb5b75ee3238b5bbe3e6815fd5650f16cd5fa0cbca6"} Oct 30 20:47:37.480982 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.480422 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" Oct 30 20:47:37.494927 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.494885 4848 generic.go:356] "Generic (PLEG): container finished" podID="84be7e6d-711c-46f8-a188-0f8955560e5a" containerID="e42ce637b917a6f0381b8b509426cd35c0a55b0c67af3c771239a94efec64213" exitCode=0 Oct 30 20:47:37.495039 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.494938 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" event={"ID":"84be7e6d-711c-46f8-a188-0f8955560e5a","Type":"ContainerDied","Data":"e42ce637b917a6f0381b8b509426cd35c0a55b0c67af3c771239a94efec64213"} Oct 30 20:47:37.495039 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.494993 4848 scope.go:122] "RemoveContainer" containerID="e42ce637b917a6f0381b8b509426cd35c0a55b0c67af3c771239a94efec64213" Oct 30 20:47:37.495364 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.495178 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" event={"ID":"84be7e6d-711c-46f8-a188-0f8955560e5a","Type":"ContainerDied","Data":"4f75a171a3bec5ccba1d2f284b771ae551435436d9c4217c9f00f234d5734276"} Oct 30 20:47:37.523732 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.523706 4848 scope.go:122] "RemoveContainer" containerID="e42ce637b917a6f0381b8b509426cd35c0a55b0c67af3c771239a94efec64213" Oct 30 20:47:37.525635 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:47:37.525613 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e42ce637b917a6f0381b8b509426cd35c0a55b0c67af3c771239a94efec64213\": container with ID starting with e42ce637b917a6f0381b8b509426cd35c0a55b0c67af3c771239a94efec64213 not found: ID does not exist" containerID="e42ce637b917a6f0381b8b509426cd35c0a55b0c67af3c771239a94efec64213" Oct 30 20:47:37.525751 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.525733 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42ce637b917a6f0381b8b509426cd35c0a55b0c67af3c771239a94efec64213"} err="failed to get container status \"e42ce637b917a6f0381b8b509426cd35c0a55b0c67af3c771239a94efec64213\": rpc error: code = NotFound desc = could not find container \"e42ce637b917a6f0381b8b509426cd35c0a55b0c67af3c771239a94efec64213\": container with ID starting with e42ce637b917a6f0381b8b509426cd35c0a55b0c67af3c771239a94efec64213 not found: ID does not exist" Oct 30 20:47:37.541641 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.541561 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" Oct 30 20:47:37.610693 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.610662 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84be7e6d-711c-46f8-a188-0f8955560e5a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84be7e6d-711c-46f8-a188-0f8955560e5a-openshift-service-ca\") pod \"84be7e6d-711c-46f8-a188-0f8955560e5a\" (UID: \"84be7e6d-711c-46f8-a188-0f8955560e5a\") " Oct 30 20:47:37.610834 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.610734 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39bb7b01-c262-4d9d-b919-74156bff3b41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39bb7b01-c262-4d9d-b919-74156bff3b41-openshift-service-ca\") pod \"39bb7b01-c262-4d9d-b919-74156bff3b41\" (UID: \"39bb7b01-c262-4d9d-b919-74156bff3b41\") " Oct 30 20:47:37.610834 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.610798 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39bb7b01-c262-4d9d-b919-74156bff3b41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39bb7b01-c262-4d9d-b919-74156bff3b41-maas-api-tls\") pod \"39bb7b01-c262-4d9d-b919-74156bff3b41\" (UID: \"39bb7b01-c262-4d9d-b919-74156bff3b41\") " Oct 30 20:47:37.610948 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.610860 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39bb7b01-c262-4d9d-b919-74156bff3b41-kube-api-access-5lk9c\" (UniqueName: \"kubernetes.io/projected/39bb7b01-c262-4d9d-b919-74156bff3b41-kube-api-access-5lk9c\") pod \"39bb7b01-c262-4d9d-b919-74156bff3b41\" (UID: \"39bb7b01-c262-4d9d-b919-74156bff3b41\") " Oct 30 20:47:37.610948 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.610898 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84be7e6d-711c-46f8-a188-0f8955560e5a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84be7e6d-711c-46f8-a188-0f8955560e5a-maas-api-tls\") pod \"84be7e6d-711c-46f8-a188-0f8955560e5a\" (UID: \"84be7e6d-711c-46f8-a188-0f8955560e5a\") " Oct 30 20:47:37.610948 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.610921 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39bb7b01-c262-4d9d-b919-74156bff3b41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/39bb7b01-c262-4d9d-b919-74156bff3b41-metrics-certs\") pod \"39bb7b01-c262-4d9d-b919-74156bff3b41\" (UID: \"39bb7b01-c262-4d9d-b919-74156bff3b41\") " Oct 30 20:47:37.611056 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.610986 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84be7e6d-711c-46f8-a188-0f8955560e5a-kube-api-access-nh8jw\" (UniqueName: \"kubernetes.io/projected/84be7e6d-711c-46f8-a188-0f8955560e5a-kube-api-access-nh8jw\") pod \"84be7e6d-711c-46f8-a188-0f8955560e5a\" (UID: \"84be7e6d-711c-46f8-a188-0f8955560e5a\") " Oct 30 20:47:37.611056 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.611011 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84be7e6d-711c-46f8-a188-0f8955560e5a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/84be7e6d-711c-46f8-a188-0f8955560e5a-metrics-certs\") pod \"84be7e6d-711c-46f8-a188-0f8955560e5a\" (UID: \"84be7e6d-711c-46f8-a188-0f8955560e5a\") " Oct 30 20:47:37.611170 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.611052 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84be7e6d-711c-46f8-a188-0f8955560e5a-openshift-service-ca" pod "84be7e6d-711c-46f8-a188-0f8955560e5a" (UID: "84be7e6d-711c-46f8-a188-0f8955560e5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:47:37.611246 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.611233 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84be7e6d-711c-46f8-a188-0f8955560e5a-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:37.611574 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.611543 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39bb7b01-c262-4d9d-b919-74156bff3b41-openshift-service-ca" pod "39bb7b01-c262-4d9d-b919-74156bff3b41" (UID: "39bb7b01-c262-4d9d-b919-74156bff3b41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:47:37.613955 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.613931 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39bb7b01-c262-4d9d-b919-74156bff3b41-maas-api-tls" pod "39bb7b01-c262-4d9d-b919-74156bff3b41" (UID: "39bb7b01-c262-4d9d-b919-74156bff3b41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:47:37.614228 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.614203 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39bb7b01-c262-4d9d-b919-74156bff3b41-metrics-certs" pod "39bb7b01-c262-4d9d-b919-74156bff3b41" (UID: "39bb7b01-c262-4d9d-b919-74156bff3b41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:47:37.614304 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.614215 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84be7e6d-711c-46f8-a188-0f8955560e5a-metrics-certs" pod "84be7e6d-711c-46f8-a188-0f8955560e5a" (UID: "84be7e6d-711c-46f8-a188-0f8955560e5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:47:37.614358 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.614332 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39bb7b01-c262-4d9d-b919-74156bff3b41-kube-api-access-5lk9c" pod "39bb7b01-c262-4d9d-b919-74156bff3b41" (UID: "39bb7b01-c262-4d9d-b919-74156bff3b41"). InnerVolumeSpecName "kube-api-access-5lk9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:37.615230 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.615211 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84be7e6d-711c-46f8-a188-0f8955560e5a-maas-api-tls" pod "84be7e6d-711c-46f8-a188-0f8955560e5a" (UID: "84be7e6d-711c-46f8-a188-0f8955560e5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:47:37.615471 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.615446 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84be7e6d-711c-46f8-a188-0f8955560e5a-kube-api-access-nh8jw" pod "84be7e6d-711c-46f8-a188-0f8955560e5a" (UID: "84be7e6d-711c-46f8-a188-0f8955560e5a"). InnerVolumeSpecName "kube-api-access-nh8jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:37.712878 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.712692 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39bb7b01-c262-4d9d-b919-74156bff3b41-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:37.712878 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.712730 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39bb7b01-c262-4d9d-b919-74156bff3b41-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:37.712878 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.712746 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lk9c\" (UniqueName: \"kubernetes.io/projected/39bb7b01-c262-4d9d-b919-74156bff3b41-kube-api-access-5lk9c\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:37.712878 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.712760 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84be7e6d-711c-46f8-a188-0f8955560e5a-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:37.712878 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.712812 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39bb7b01-c262-4d9d-b919-74156bff3b41-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:37.712878 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.712828 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh8jw\" (UniqueName: \"kubernetes.io/projected/84be7e6d-711c-46f8-a188-0f8955560e5a-kube-api-access-nh8jw\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:37.712878 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.712841 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84be7e6d-711c-46f8-a188-0f8955560e5a-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:37.729837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.729802 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw"] Oct 30 20:47:37.730293 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.730271 4848 cpu_manager.go:414] "containerMap: removing container" podUID="84be7e6d-711c-46f8-a188-0f8955560e5a" containerName="maas-api" Oct 30 20:47:37.730293 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.730290 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84be7e6d-711c-46f8-a188-0f8955560e5a" containerName="maas-api" Oct 30 20:47:37.730293 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.730300 4848 cpu_manager.go:414] "containerMap: removing container" podUID="39bb7b01-c262-4d9d-b919-74156bff3b41" containerName="maas-api" Oct 30 20:47:37.730527 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.730305 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39bb7b01-c262-4d9d-b919-74156bff3b41" containerName="maas-api" Oct 30 20:47:37.730527 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.730376 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="84be7e6d-711c-46f8-a188-0f8955560e5a" containerName="maas-api" Oct 30 20:47:37.730527 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.730383 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="39bb7b01-c262-4d9d-b919-74156bff3b41" containerName="maas-api" Oct 30 20:47:37.735019 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.734994 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" Oct 30 20:47:37.739985 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.739961 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-f9a819-serving-cert\"" Oct 30 20:47:37.751494 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.751466 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw"] Oct 30 20:47:37.814224 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.814138 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3aee534-70d3-4567-8a70-063a4e684526-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw\" (UID: \"a3aee534-70d3-4567-8a70-063a4e684526\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" Oct 30 20:47:37.814367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.814234 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3aee534-70d3-4567-8a70-063a4e684526-metrics-certs\") pod \"maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw\" (UID: \"a3aee534-70d3-4567-8a70-063a4e684526\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" Oct 30 20:47:37.814367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.814329 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3aee534-70d3-4567-8a70-063a4e684526-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw\" (UID: \"a3aee534-70d3-4567-8a70-063a4e684526\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" Oct 30 20:47:37.814453 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.814372 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7t9z\" (UniqueName: \"kubernetes.io/projected/a3aee534-70d3-4567-8a70-063a4e684526-kube-api-access-x7t9z\") pod \"maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw\" (UID: \"a3aee534-70d3-4567-8a70-063a4e684526\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" Oct 30 20:47:37.915271 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.915239 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3aee534-70d3-4567-8a70-063a4e684526-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw\" (UID: \"a3aee534-70d3-4567-8a70-063a4e684526\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" Oct 30 20:47:37.915466 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.915281 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7t9z\" (UniqueName: \"kubernetes.io/projected/a3aee534-70d3-4567-8a70-063a4e684526-kube-api-access-x7t9z\") pod \"maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw\" (UID: \"a3aee534-70d3-4567-8a70-063a4e684526\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" Oct 30 20:47:37.915466 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.915328 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3aee534-70d3-4567-8a70-063a4e684526-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw\" (UID: \"a3aee534-70d3-4567-8a70-063a4e684526\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" Oct 30 20:47:37.915466 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.915412 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3aee534-70d3-4567-8a70-063a4e684526-metrics-certs\") pod \"maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw\" (UID: \"a3aee534-70d3-4567-8a70-063a4e684526\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" Oct 30 20:47:37.916065 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.916042 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3aee534-70d3-4567-8a70-063a4e684526-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw\" (UID: \"a3aee534-70d3-4567-8a70-063a4e684526\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" Oct 30 20:47:37.917894 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.917872 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3aee534-70d3-4567-8a70-063a4e684526-maas-api-tls\") pod \"maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw\" (UID: \"a3aee534-70d3-4567-8a70-063a4e684526\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" Oct 30 20:47:37.917894 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.917891 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3aee534-70d3-4567-8a70-063a4e684526-metrics-certs\") pod \"maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw\" (UID: \"a3aee534-70d3-4567-8a70-063a4e684526\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" Oct 30 20:47:37.926279 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:37.926258 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7t9z\" (UniqueName: \"kubernetes.io/projected/a3aee534-70d3-4567-8a70-063a4e684526-kube-api-access-x7t9z\") pod \"maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw\" (UID: \"a3aee534-70d3-4567-8a70-063a4e684526\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" Oct 30 20:47:38.049403 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.049373 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" Oct 30 20:47:38.334624 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.334552 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw"] Oct 30 20:47:38.336900 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:47:38.336873 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3aee534_70d3_4567_8a70_063a4e684526.slice/crio-41245cc0a7eee2a8d7e14f53d22c8d0bbc08a0a366dd6b41d2c9da74ca7165e0 WatchSource:0}: Error finding container 41245cc0a7eee2a8d7e14f53d22c8d0bbc08a0a366dd6b41d2c9da74ca7165e0: Status 404 returned error can't find the container with id 41245cc0a7eee2a8d7e14f53d22c8d0bbc08a0a366dd6b41d2c9da74ca7165e0 Oct 30 20:47:38.502097 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.502051 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" event={"ID":"9f7e1ecd-c614-4687-96e3-631e5716cf83","Type":"ContainerStarted","Data":"2257b757aa8b729d0f2132a0da342937e3b711820c1b9b2a7644e8dd32f9b92e"} Oct 30 20:47:38.502620 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.502144 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" Oct 30 20:47:38.503716 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.503688 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr" event={"ID":"6867e706-6187-4821-ae3c-141f443718a5","Type":"ContainerStarted","Data":"ba0c27980a493c24c11f878bce0f8fde72b2cb67b173c248d6b2b8377f93d3cd"} Oct 30 20:47:38.503867 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.503754 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr" Oct 30 20:47:38.505100 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.505084 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" Oct 30 20:47:38.505195 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.505102 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz" event={"ID":"39bb7b01-c262-4d9d-b919-74156bff3b41","Type":"ContainerDied","Data":"87dcd2cbc04b6a26badf04b234b56eda3535729c85d73da620c04f19f726cdfc"} Oct 30 20:47:38.505195 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.505148 4848 scope.go:122] "RemoveContainer" containerID="fb00f5792fd61211a8cfeeb5b75ee3238b5bbe3e6815fd5650f16cd5fa0cbca6" Oct 30 20:47:38.506364 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.506082 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld" Oct 30 20:47:38.514636 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.514604 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" event={"ID":"a3aee534-70d3-4567-8a70-063a4e684526","Type":"ContainerStarted","Data":"41245cc0a7eee2a8d7e14f53d22c8d0bbc08a0a366dd6b41d2c9da74ca7165e0"} Oct 30 20:47:38.516414 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.516380 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d" event={"ID":"3d385f07-6dae-4ca6-9e02-eb587a7d6374","Type":"ContainerStarted","Data":"53571241600ecbf7f3e5f0e47a2a7f5173a59897a2988b637da5af2f2d94e164"} Oct 30 20:47:38.516484 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.516436 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d" event={"ID":"3d385f07-6dae-4ca6-9e02-eb587a7d6374","Type":"ContainerStarted","Data":"b6ef1f234a7febcf833c130e1081bca0076f21d5016db96e7e4bda61d093b8fc"} Oct 30 20:47:38.516484 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.516475 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d" Oct 30 20:47:38.531876 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.531830 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" podStartSLOduration=2.040675919 podStartE2EDuration="2.531818075s" totalImagesPullingTime="491.142156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:38.529365618 +0000 UTC m=+563.709262125" watchObservedRunningTime="2026-10-30 20:47:38.531818075 +0000 UTC m=+563.711714610" Oct 30 20:47:38.547044 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.547015 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld"] Oct 30 20:47:38.551862 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.551837 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-eaf151e7-b88c77cc5-qxdld"] Oct 30 20:47:38.568053 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.567999 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr" podStartSLOduration=2.567984964 podStartE2EDuration="2.567984964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:38.566110701 +0000 UTC m=+563.746007208" watchObservedRunningTime="2026-10-30 20:47:38.567984964 +0000 UTC m=+563.747881470" Oct 30 20:47:38.581795 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.581751 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz"] Oct 30 20:47:38.587928 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.587896 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67598f1d-54d65f9857-dlsqz"] Oct 30 20:47:38.950685 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.950646 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39bb7b01-c262-4d9d-b919-74156bff3b41" path="/var/lib/kubelet/pods/39bb7b01-c262-4d9d-b919-74156bff3b41/volumes" Oct 30 20:47:38.951201 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:38.951175 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84be7e6d-711c-46f8-a188-0f8955560e5a" path="/var/lib/kubelet/pods/84be7e6d-711c-46f8-a188-0f8955560e5a/volumes" Oct 30 20:47:39.454556 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:39.454517 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5" podUID="1f1aae51-6213-467c-a0ef-26f5fca2bdef" containerName="payload-processing" containerID="cri-o://18f62328bb0589c1343c34783610e7ff898b5c06b7da262a0bfbe3335c9821a2" gracePeriod=25 Oct 30 20:47:39.524434 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:39.524401 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" event={"ID":"a3aee534-70d3-4567-8a70-063a4e684526","Type":"ContainerStarted","Data":"380ee59b786cee0efee72b3fa65f1e8e0f87218a52cc80f9ca40456314c1f884"} Oct 30 20:47:39.524893 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:39.524474 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" Oct 30 20:47:39.554350 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:39.554229 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" podStartSLOduration=2.166263052 podStartE2EDuration="2.554210411s" totalImagesPullingTime="387.947359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:39.553230049 +0000 UTC m=+564.733126558" watchObservedRunningTime="2026-10-30 20:47:39.554210411 +0000 UTC m=+564.734106919" Oct 30 20:47:39.566744 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:39.566674 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d" podStartSLOduration=3.56665706 podStartE2EDuration="3.56665706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:38.604179566 +0000 UTC m=+563.784076073" watchObservedRunningTime="2026-10-30 20:47:39.56665706 +0000 UTC m=+564.746553568" Oct 30 20:47:39.741681 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:39.741650 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5" Oct 30 20:47:39.855712 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:39.855677 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1aae51-6213-467c-a0ef-26f5fca2bdef-kube-api-access-wlfr8\" (UniqueName: \"kubernetes.io/projected/1f1aae51-6213-467c-a0ef-26f5fca2bdef-kube-api-access-wlfr8\") pod \"1f1aae51-6213-467c-a0ef-26f5fca2bdef\" (UID: \"1f1aae51-6213-467c-a0ef-26f5fca2bdef\") " Oct 30 20:47:39.855905 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:39.855723 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1aae51-6213-467c-a0ef-26f5fca2bdef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f1aae51-6213-467c-a0ef-26f5fca2bdef-plugins-config-volume\") pod \"1f1aae51-6213-467c-a0ef-26f5fca2bdef\" (UID: \"1f1aae51-6213-467c-a0ef-26f5fca2bdef\") " Oct 30 20:47:39.856106 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:39.856084 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1aae51-6213-467c-a0ef-26f5fca2bdef-plugins-config-volume" pod "1f1aae51-6213-467c-a0ef-26f5fca2bdef" (UID: "1f1aae51-6213-467c-a0ef-26f5fca2bdef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:47:39.857754 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:39.857728 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1aae51-6213-467c-a0ef-26f5fca2bdef-kube-api-access-wlfr8" pod "1f1aae51-6213-467c-a0ef-26f5fca2bdef" (UID: "1f1aae51-6213-467c-a0ef-26f5fca2bdef"). InnerVolumeSpecName "kube-api-access-wlfr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:39.956877 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:39.956848 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlfr8\" (UniqueName: \"kubernetes.io/projected/1f1aae51-6213-467c-a0ef-26f5fca2bdef-kube-api-access-wlfr8\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:39.956877 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:39.956875 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f1aae51-6213-467c-a0ef-26f5fca2bdef-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:40.534055 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.534023 4848 generic.go:356] "Generic (PLEG): container finished" podID="1f1aae51-6213-467c-a0ef-26f5fca2bdef" containerID="18f62328bb0589c1343c34783610e7ff898b5c06b7da262a0bfbe3335c9821a2" exitCode=0 Oct 30 20:47:40.534553 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.534070 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5" Oct 30 20:47:40.534553 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.534104 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5" event={"ID":"1f1aae51-6213-467c-a0ef-26f5fca2bdef","Type":"ContainerDied","Data":"18f62328bb0589c1343c34783610e7ff898b5c06b7da262a0bfbe3335c9821a2"} Oct 30 20:47:40.534553 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.534164 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5" event={"ID":"1f1aae51-6213-467c-a0ef-26f5fca2bdef","Type":"ContainerDied","Data":"41a7324393364c20037b35a9d457fae829b4431529e42a7c3bfce87d38c03572"} Oct 30 20:47:40.534553 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.534196 4848 scope.go:122] "RemoveContainer" containerID="18f62328bb0589c1343c34783610e7ff898b5c06b7da262a0bfbe3335c9821a2" Oct 30 20:47:40.543359 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.543342 4848 scope.go:122] "RemoveContainer" containerID="18f62328bb0589c1343c34783610e7ff898b5c06b7da262a0bfbe3335c9821a2" Oct 30 20:47:40.543617 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:47:40.543600 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f62328bb0589c1343c34783610e7ff898b5c06b7da262a0bfbe3335c9821a2\": container with ID starting with 18f62328bb0589c1343c34783610e7ff898b5c06b7da262a0bfbe3335c9821a2 not found: ID does not exist" containerID="18f62328bb0589c1343c34783610e7ff898b5c06b7da262a0bfbe3335c9821a2" Oct 30 20:47:40.543659 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.543626 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f62328bb0589c1343c34783610e7ff898b5c06b7da262a0bfbe3335c9821a2"} err="failed to get container status \"18f62328bb0589c1343c34783610e7ff898b5c06b7da262a0bfbe3335c9821a2\": rpc error: code = NotFound desc = could not find container \"18f62328bb0589c1343c34783610e7ff898b5c06b7da262a0bfbe3335c9821a2\": container with ID starting with 18f62328bb0589c1343c34783610e7ff898b5c06b7da262a0bfbe3335c9821a2 not found: ID does not exist" Oct 30 20:47:40.561154 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.561103 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5"] Oct 30 20:47:40.563521 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.563498 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b16c21be-c777df947-zkld5"] Oct 30 20:47:40.669133 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.666878 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g"] Oct 30 20:47:40.669133 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.667354 4848 cpu_manager.go:414] "containerMap: removing container" podUID="1f1aae51-6213-467c-a0ef-26f5fca2bdef" containerName="payload-processing" Oct 30 20:47:40.669133 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.667366 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1aae51-6213-467c-a0ef-26f5fca2bdef" containerName="payload-processing" Oct 30 20:47:40.669133 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.667433 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1aae51-6213-467c-a0ef-26f5fca2bdef" containerName="payload-processing" Oct 30 20:47:40.672205 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.672071 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.679817 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.679076 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-bc102e-openshift-default-dockercfg-wm7tq\"" Oct 30 20:47:40.686461 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.686431 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g"] Oct 30 20:47:40.765573 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.765540 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.765573 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.765576 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.765870 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.765599 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.765870 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.765746 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.765870 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.765851 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.766013 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.765888 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.766013 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.765922 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.766013 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.765990 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdbmq\" (UniqueName: \"kubernetes.io/projected/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-kube-api-access-zdbmq\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.766177 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.766035 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.867642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.867074 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.867642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.867128 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.867642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.867166 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.867642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.867212 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.867642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.867280 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.867642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.867308 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.867642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.867341 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.867642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.867388 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdbmq\" (UniqueName: \"kubernetes.io/projected/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-kube-api-access-zdbmq\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.867642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.867430 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.868140 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.867957 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.868615 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.868590 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.871281 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.871256 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.871530 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.871511 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.872052 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.872020 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.872172 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.872129 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.872564 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.872543 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.880688 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.880657 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.882793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.882737 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdbmq\" (UniqueName: \"kubernetes.io/projected/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-kube-api-access-zdbmq\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:40.936402 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.935963 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1aae51-6213-467c-a0ef-26f5fca2bdef" path="/var/lib/kubelet/pods/1f1aae51-6213-467c-a0ef-26f5fca2bdef/volumes" Oct 30 20:47:40.988252 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:40.988220 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:41.297639 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:41.297608 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g"] Oct 30 20:47:41.299830 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:47:41.299758 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bdf57f3_fcef_4f4c_82b3_3b8ce7e3882b.slice/crio-38f833e4d0f3a1b6954dfc06964978459596beb5372fb0867cf0a4e82e2e5529 WatchSource:0}: Error finding container 38f833e4d0f3a1b6954dfc06964978459596beb5372fb0867cf0a4e82e2e5529: Status 404 returned error can't find the container with id 38f833e4d0f3a1b6954dfc06964978459596beb5372fb0867cf0a4e82e2e5529 Oct 30 20:47:41.302471 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:41.302441 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:41.302558 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:41.302523 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:41.302615 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:41.302571 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:41.542250 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:41.542214 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" event={"ID":"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b","Type":"ContainerStarted","Data":"841cbf266602699d8a58d63d0a2d6ceea73042b1bf5a9a829ca16cb7c87629dd"} Oct 30 20:47:41.542819 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:41.542279 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" event={"ID":"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b","Type":"ContainerStarted","Data":"38f833e4d0f3a1b6954dfc06964978459596beb5372fb0867cf0a4e82e2e5529"} Oct 30 20:47:41.564011 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:41.563907 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" podStartSLOduration=1.563892472 podStartE2EDuration="1.563892472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:41.562866209 +0000 UTC m=+566.742762719" watchObservedRunningTime="2026-10-30 20:47:41.563892472 +0000 UTC m=+566.743788978" Oct 30 20:47:41.882213 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:41.882132 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g" podUID="91767111-f078-4f24-88dd-fd4ee52fd113" containerName="payload-pre-processing" containerID="cri-o://7f81c6a4831383c026202f7956cf17e0950a911225407b1c268ab2125dfd4943" gracePeriod=25 Oct 30 20:47:41.989912 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:41.989839 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:42.366999 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:42.366938 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g" Oct 30 20:47:42.492123 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:42.492034 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91767111-f078-4f24-88dd-fd4ee52fd113-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91767111-f078-4f24-88dd-fd4ee52fd113-plugins-config-volume\") pod \"91767111-f078-4f24-88dd-fd4ee52fd113\" (UID: \"91767111-f078-4f24-88dd-fd4ee52fd113\") " Oct 30 20:47:42.492301 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:42.492128 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91767111-f078-4f24-88dd-fd4ee52fd113-kube-api-access-6wcvl\" (UniqueName: \"kubernetes.io/projected/91767111-f078-4f24-88dd-fd4ee52fd113-kube-api-access-6wcvl\") pod \"91767111-f078-4f24-88dd-fd4ee52fd113\" (UID: \"91767111-f078-4f24-88dd-fd4ee52fd113\") " Oct 30 20:47:42.492423 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:42.492387 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91767111-f078-4f24-88dd-fd4ee52fd113-plugins-config-volume" pod "91767111-f078-4f24-88dd-fd4ee52fd113" (UID: "91767111-f078-4f24-88dd-fd4ee52fd113"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:47:42.492543 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:42.492510 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91767111-f078-4f24-88dd-fd4ee52fd113-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:42.494282 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:42.494253 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91767111-f078-4f24-88dd-fd4ee52fd113-kube-api-access-6wcvl" pod "91767111-f078-4f24-88dd-fd4ee52fd113" (UID: "91767111-f078-4f24-88dd-fd4ee52fd113"). InnerVolumeSpecName "kube-api-access-6wcvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:47:42.550853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:42.550814 4848 generic.go:356] "Generic (PLEG): container finished" podID="91767111-f078-4f24-88dd-fd4ee52fd113" containerID="7f81c6a4831383c026202f7956cf17e0950a911225407b1c268ab2125dfd4943" exitCode=0 Oct 30 20:47:42.551305 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:42.550863 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g" Oct 30 20:47:42.551305 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:42.550899 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g" event={"ID":"91767111-f078-4f24-88dd-fd4ee52fd113","Type":"ContainerDied","Data":"7f81c6a4831383c026202f7956cf17e0950a911225407b1c268ab2125dfd4943"} Oct 30 20:47:42.551305 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:42.550962 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g" event={"ID":"91767111-f078-4f24-88dd-fd4ee52fd113","Type":"ContainerDied","Data":"81b0b364e44ed66089277b435ce78f2b78d117565e92972df7f84169626f86f2"} Oct 30 20:47:42.551305 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:42.550994 4848 scope.go:122] "RemoveContainer" containerID="7f81c6a4831383c026202f7956cf17e0950a911225407b1c268ab2125dfd4943" Oct 30 20:47:42.565060 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:42.562399 4848 scope.go:122] "RemoveContainer" containerID="7f81c6a4831383c026202f7956cf17e0950a911225407b1c268ab2125dfd4943" Oct 30 20:47:42.565060 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:47:42.562813 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f81c6a4831383c026202f7956cf17e0950a911225407b1c268ab2125dfd4943\": container with ID starting with 7f81c6a4831383c026202f7956cf17e0950a911225407b1c268ab2125dfd4943 not found: ID does not exist" containerID="7f81c6a4831383c026202f7956cf17e0950a911225407b1c268ab2125dfd4943" Oct 30 20:47:42.565060 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:42.562895 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f81c6a4831383c026202f7956cf17e0950a911225407b1c268ab2125dfd4943"} err="failed to get container status \"7f81c6a4831383c026202f7956cf17e0950a911225407b1c268ab2125dfd4943\": rpc error: code = NotFound desc = could not find container \"7f81c6a4831383c026202f7956cf17e0950a911225407b1c268ab2125dfd4943\": container with ID starting with 7f81c6a4831383c026202f7956cf17e0950a911225407b1c268ab2125dfd4943 not found: ID does not exist" Oct 30 20:47:42.590567 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:42.590537 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g"] Oct 30 20:47:42.593458 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:42.593434 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wcvl\" (UniqueName: \"kubernetes.io/projected/91767111-f078-4f24-88dd-fd4ee52fd113-kube-api-access-6wcvl\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:47:42.636472 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:42.636398 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67598f1d-7cb658879-swx2g"] Oct 30 20:47:42.936575 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:42.936546 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91767111-f078-4f24-88dd-fd4ee52fd113" path="/var/lib/kubelet/pods/91767111-f078-4f24-88dd-fd4ee52fd113/volumes" Oct 30 20:47:42.994103 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:42.994075 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:43.565927 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:43.565817 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:43.567342 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:43.567306 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:47:44.549031 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:44.548988 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" Oct 30 20:47:45.558631 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:45.558600 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" Oct 30 20:47:45.662048 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:45.661985 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86"] Oct 30 20:47:45.663287 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:45.663260 4848 cpu_manager.go:414] "containerMap: removing container" podUID="91767111-f078-4f24-88dd-fd4ee52fd113" containerName="payload-pre-processing" Oct 30 20:47:45.663430 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:45.663297 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91767111-f078-4f24-88dd-fd4ee52fd113" containerName="payload-pre-processing" Oct 30 20:47:45.663490 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:45.663454 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="91767111-f078-4f24-88dd-fd4ee52fd113" containerName="payload-pre-processing" Oct 30 20:47:45.668408 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:45.668374 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86"] Oct 30 20:47:45.668514 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:45.668472 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86" Oct 30 20:47:45.689832 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:45.688867 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-bc102e-dockercfg-pvrvc\"" Oct 30 20:47:45.689986 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:45.689850 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-bc102e\"" Oct 30 20:47:45.784787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:45.780469 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2txq\" (UniqueName: \"kubernetes.io/projected/b7f2555a-89d9-4d58-89d5-f076b60af465-kube-api-access-w2txq\") pod \"payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86\" (UID: \"b7f2555a-89d9-4d58-89d5-f076b60af465\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86" Oct 30 20:47:45.784787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:45.780589 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f2555a-89d9-4d58-89d5-f076b60af465-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86\" (UID: \"b7f2555a-89d9-4d58-89d5-f076b60af465\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86" Oct 30 20:47:45.882018 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:45.881982 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f2555a-89d9-4d58-89d5-f076b60af465-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86\" (UID: \"b7f2555a-89d9-4d58-89d5-f076b60af465\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86" Oct 30 20:47:45.882167 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:45.882125 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2txq\" (UniqueName: \"kubernetes.io/projected/b7f2555a-89d9-4d58-89d5-f076b60af465-kube-api-access-w2txq\") pod \"payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86\" (UID: \"b7f2555a-89d9-4d58-89d5-f076b60af465\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86" Oct 30 20:47:45.882697 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:45.882673 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f2555a-89d9-4d58-89d5-f076b60af465-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86\" (UID: \"b7f2555a-89d9-4d58-89d5-f076b60af465\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86" Oct 30 20:47:45.897440 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:45.897407 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2txq\" (UniqueName: \"kubernetes.io/projected/b7f2555a-89d9-4d58-89d5-f076b60af465-kube-api-access-w2txq\") pod \"payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86\" (UID: \"b7f2555a-89d9-4d58-89d5-f076b60af465\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86" Oct 30 20:47:45.987350 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:45.987319 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86" Oct 30 20:47:46.344789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:46.323014 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86"] Oct 30 20:47:46.582340 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:46.582307 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86" event={"ID":"b7f2555a-89d9-4d58-89d5-f076b60af465","Type":"ContainerStarted","Data":"272be5d67b61f9b848032198b237badf12d5794f07a997f8245a6ddc98fcdac5"} Oct 30 20:47:46.582988 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:46.582357 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86" event={"ID":"b7f2555a-89d9-4d58-89d5-f076b60af465","Type":"ContainerStarted","Data":"8b65909440080e24039ad39336b098828710f5585e5b304506568a8d143fe49a"} Oct 30 20:47:46.582988 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:46.582397 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86" Oct 30 20:47:46.614879 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:46.611062 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86" podStartSLOduration=1.611042713 podStartE2EDuration="1.611042713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:46.600126873 +0000 UTC m=+571.780023380" watchObservedRunningTime="2026-10-30 20:47:46.611042713 +0000 UTC m=+571.790939222" Oct 30 20:47:47.429793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:47.417438 4848 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:47:47.429793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:47.417500 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" podUID="e57add2c-5ccf-47eb-8adb-d5852289b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:49.527923 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:49.527891 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d" Oct 30 20:47:49.528413 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:49.527971 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr" Oct 30 20:47:50.424928 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:50.424893 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:47:50.425152 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:50.424956 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" podUID="8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:47:50.610661 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:50.610631 4848 generic.go:356] "Generic (PLEG): container finished" podID="e57add2c-5ccf-47eb-8adb-d5852289b111" containerID="8e74ab0c9e3f60b54bdd7640b98965b377814024591e577b3ef4af4a82d75b86" exitCode=0 Oct 30 20:47:50.611060 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:50.610710 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" event={"ID":"e57add2c-5ccf-47eb-8adb-d5852289b111","Type":"ContainerDied","Data":"8e74ab0c9e3f60b54bdd7640b98965b377814024591e577b3ef4af4a82d75b86"} Oct 30 20:47:50.611226 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:50.611209 4848 scope.go:122] "RemoveContainer" containerID="8e74ab0c9e3f60b54bdd7640b98965b377814024591e577b3ef4af4a82d75b86" Oct 30 20:47:50.612582 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:50.612558 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:50.612650 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:50.612621 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:50.612688 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:50.612655 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:51.647555 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:51.647520 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" event={"ID":"e57add2c-5ccf-47eb-8adb-d5852289b111","Type":"ContainerStarted","Data":"5ffdf4d18fd52c60a0ac175d0f926866aaba4fb54478651b05a380dd5c413d07"} Oct 30 20:47:52.292825 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.290451 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz"] Oct 30 20:47:52.293857 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.293048 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" Oct 30 20:47:52.300851 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.300820 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-8e4acb-serving-cert\"" Oct 30 20:47:52.349802 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.349170 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz"] Oct 30 20:47:52.393451 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.393417 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4nph\" (UniqueName: \"kubernetes.io/projected/7d33a513-4751-4c10-bbbe-061498a405f3-kube-api-access-x4nph\") pod \"maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz\" (UID: \"7d33a513-4751-4c10-bbbe-061498a405f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" Oct 30 20:47:52.393451 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.393452 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d33a513-4751-4c10-bbbe-061498a405f3-maas-api-tls\") pod \"maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz\" (UID: \"7d33a513-4751-4c10-bbbe-061498a405f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" Oct 30 20:47:52.393635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.393474 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d33a513-4751-4c10-bbbe-061498a405f3-metrics-certs\") pod \"maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz\" (UID: \"7d33a513-4751-4c10-bbbe-061498a405f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" Oct 30 20:47:52.393635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.393604 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d33a513-4751-4c10-bbbe-061498a405f3-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz\" (UID: \"7d33a513-4751-4c10-bbbe-061498a405f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" Oct 30 20:47:52.427792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.425177 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:52.427792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.425311 4848 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:47:52.427792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.425354 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" podUID="e57add2c-5ccf-47eb-8adb-d5852289b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 20:47:52.494951 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.494841 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4nph\" (UniqueName: \"kubernetes.io/projected/7d33a513-4751-4c10-bbbe-061498a405f3-kube-api-access-x4nph\") pod \"maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz\" (UID: \"7d33a513-4751-4c10-bbbe-061498a405f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" Oct 30 20:47:52.494951 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.494902 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d33a513-4751-4c10-bbbe-061498a405f3-maas-api-tls\") pod \"maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz\" (UID: \"7d33a513-4751-4c10-bbbe-061498a405f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" Oct 30 20:47:52.494951 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.494945 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d33a513-4751-4c10-bbbe-061498a405f3-metrics-certs\") pod \"maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz\" (UID: \"7d33a513-4751-4c10-bbbe-061498a405f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" Oct 30 20:47:52.495252 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.495038 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d33a513-4751-4c10-bbbe-061498a405f3-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz\" (UID: \"7d33a513-4751-4c10-bbbe-061498a405f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" Oct 30 20:47:52.497381 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.496056 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d33a513-4751-4c10-bbbe-061498a405f3-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz\" (UID: \"7d33a513-4751-4c10-bbbe-061498a405f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" Oct 30 20:47:52.498079 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.498052 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d33a513-4751-4c10-bbbe-061498a405f3-metrics-certs\") pod \"maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz\" (UID: \"7d33a513-4751-4c10-bbbe-061498a405f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" Oct 30 20:47:52.498185 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.498084 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d33a513-4751-4c10-bbbe-061498a405f3-maas-api-tls\") pod \"maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz\" (UID: \"7d33a513-4751-4c10-bbbe-061498a405f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" Oct 30 20:47:52.505838 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.504616 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4nph\" (UniqueName: \"kubernetes.io/projected/7d33a513-4751-4c10-bbbe-061498a405f3-kube-api-access-x4nph\") pod \"maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz\" (UID: \"7d33a513-4751-4c10-bbbe-061498a405f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" Oct 30 20:47:52.535925 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:47:52.534730 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:47:52.648596 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.648504 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" Oct 30 20:47:52.654728 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.654698 4848 generic.go:356] "Generic (PLEG): container finished" podID="8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" containerID="fa2a27fc7a0dc4cf57ca23ba474f5227396d6e7d6ad364b24dd0edbbc205e11f" exitCode=0 Oct 30 20:47:52.654870 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.654786 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" event={"ID":"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4","Type":"ContainerDied","Data":"fa2a27fc7a0dc4cf57ca23ba474f5227396d6e7d6ad364b24dd0edbbc205e11f"} Oct 30 20:47:52.655278 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.655255 4848 scope.go:122] "RemoveContainer" containerID="fa2a27fc7a0dc4cf57ca23ba474f5227396d6e7d6ad364b24dd0edbbc205e11f" Oct 30 20:47:52.658007 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.657976 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:52.658076 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.658047 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:52.658114 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.658082 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:47:52.942131 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:47:52.941978 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d33a513_4751_4c10_bbbe_061498a405f3.slice/crio-7d0986c290324f9c040c3840194133745072963fe401450157d41a0d9ce8eeba WatchSource:0}: Error finding container 7d0986c290324f9c040c3840194133745072963fe401450157d41a0d9ce8eeba: Status 404 returned error can't find the container with id 7d0986c290324f9c040c3840194133745072963fe401450157d41a0d9ce8eeba Oct 30 20:47:52.944010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:52.943967 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz"] Oct 30 20:47:53.434843 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:53.427259 4848 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:47:53.434843 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:53.427330 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" podUID="e57add2c-5ccf-47eb-8adb-d5852289b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 20:47:53.692927 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:53.692644 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" event={"ID":"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4","Type":"ContainerStarted","Data":"7bdf73f56e3d4e0a2c68d166ed6927dd0a13e73b9a3ab980fd678cf9c576b753"} Oct 30 20:47:53.694373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:53.694349 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" event={"ID":"7d33a513-4751-4c10-bbbe-061498a405f3","Type":"ContainerStarted","Data":"7d0986c290324f9c040c3840194133745072963fe401450157d41a0d9ce8eeba"} Oct 30 20:47:54.424792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:54.421292 4848 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:47:54.424792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:54.421374 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" podUID="e57add2c-5ccf-47eb-8adb-d5852289b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 20:47:54.671168 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:54.671133 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:54.671548 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:54.671443 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 20:47:54.671732 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:54.671707 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" podUID="8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 20:47:54.701571 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:54.701491 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" event={"ID":"7d33a513-4751-4c10-bbbe-061498a405f3","Type":"ContainerStarted","Data":"26888debaa6c97f9fc70b6e5a3b0e018bdcd0dd51401000d5e9c164f6f388625"} Oct 30 20:47:54.702216 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:54.702029 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" Oct 30 20:47:54.751789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:54.749050 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" podStartSLOduration=2.201184465 podStartE2EDuration="2.749028699s" totalImagesPullingTime="547.844234ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:47:54.747015164 +0000 UTC m=+579.926911676" watchObservedRunningTime="2026-10-30 20:47:54.749028699 +0000 UTC m=+579.928925206" Oct 30 20:47:55.421553 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:55.421125 4848 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:47:55.421553 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:55.421183 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" podUID="e57add2c-5ccf-47eb-8adb-d5852289b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 20:47:55.677798 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:55.677350 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 20:47:55.677798 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:55.677428 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" podUID="8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 20:47:56.426791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:56.426230 4848 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 30 20:47:56.426791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:56.426307 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" podUID="e57add2c-5ccf-47eb-8adb-d5852289b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 30 20:47:56.681875 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:56.681758 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 20:47:56.682138 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:56.682109 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" podUID="8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 20:47:57.436790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:57.436058 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:57.440789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:57.437394 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:57.459419 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:57.457056 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 20:47:57.592015 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:57.591988 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86" Oct 30 20:47:57.669374 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:57.669339 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 20:47:57.669658 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:57.669625 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" podUID="8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 20:47:58.669290 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:58.669246 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 30 20:47:58.669684 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:58.669313 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" podUID="8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 30 20:47:59.672914 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:59.672888 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:59.673374 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:59.673188 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:47:59.684038 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:47:59.684007 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 20:48:01.746962 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:01.746930 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" Oct 30 20:48:02.801593 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:02.801561 4848 generic.go:356] "Generic (PLEG): container finished" podID="0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" containerID="841cbf266602699d8a58d63d0a2d6ceea73042b1bf5a9a829ca16cb7c87629dd" exitCode=0 Oct 30 20:48:02.802069 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:02.801633 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" event={"ID":"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b","Type":"ContainerDied","Data":"841cbf266602699d8a58d63d0a2d6ceea73042b1bf5a9a829ca16cb7c87629dd"} Oct 30 20:48:02.802069 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:02.802060 4848 scope.go:122] "RemoveContainer" containerID="841cbf266602699d8a58d63d0a2d6ceea73042b1bf5a9a829ca16cb7c87629dd" Oct 30 20:48:02.803348 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:02.803322 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:48:02.803423 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:02.803393 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:48:02.803465 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:02.803424 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:48:03.832790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:03.832474 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" event={"ID":"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b","Type":"ContainerStarted","Data":"3374da1831c1b0d991f7b729419dc88a294f7c1a34e01b5d025ad772ff54618a"} Oct 30 20:48:03.989102 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:03.989068 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:48:03.989272 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:03.989248 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 20:48:03.989320 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:03.989302 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" podUID="0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 20:48:04.989495 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:04.989465 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 20:48:04.989934 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:04.989525 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" podUID="0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 20:48:05.989455 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:05.989423 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 20:48:05.989667 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:05.989472 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" podUID="0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 20:48:06.989568 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:06.989534 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 20:48:06.989746 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:06.989606 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" podUID="0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 20:48:07.989102 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:07.989062 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 20:48:07.989301 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:07.989119 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" podUID="0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 20:48:08.992188 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:08.992158 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:48:08.992657 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:08.992504 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:48:08.993184 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:08.993167 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 20:48:09.933954 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:09.933924 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt"] Oct 30 20:48:09.937248 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:09.937227 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:09.947546 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:09.947518 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-bc102e\"/\"kube-root-ca.crt\"" Oct 30 20:48:09.947858 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:09.947842 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-bc102e\"/\"openshift-service-ca.crt\"" Oct 30 20:48:09.949963 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:09.949944 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-bc102e\"/\"default-dockercfg-s2c6d\"" Oct 30 20:48:09.950066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:09.950033 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt"] Oct 30 20:48:09.950131 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:09.950067 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-bc102e\"/\"simulator-w0-bc102e-kserve-self-signed-certs\"" Oct 30 20:48:10.000886 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.000856 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqrjg\" (UniqueName: \"kubernetes.io/projected/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-kube-api-access-pqrjg\") pod \"simulator-w0-bc102e-kserve-798dd5948b-7tnxt\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:10.001215 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.000888 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-model-cache\") pod \"simulator-w0-bc102e-kserve-798dd5948b-7tnxt\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:10.001215 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.001012 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-dshm\") pod \"simulator-w0-bc102e-kserve-798dd5948b-7tnxt\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:10.001215 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.001053 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-home\") pod \"simulator-w0-bc102e-kserve-798dd5948b-7tnxt\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:10.001215 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.001108 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-tmp-dir\") pod \"simulator-w0-bc102e-kserve-798dd5948b-7tnxt\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:10.001215 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.001144 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-tls-certs\") pod \"simulator-w0-bc102e-kserve-798dd5948b-7tnxt\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:10.103613 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.103569 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqrjg\" (UniqueName: \"kubernetes.io/projected/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-kube-api-access-pqrjg\") pod \"simulator-w0-bc102e-kserve-798dd5948b-7tnxt\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:10.104255 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.104227 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-model-cache\") pod \"simulator-w0-bc102e-kserve-798dd5948b-7tnxt\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:10.105015 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.104994 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-dshm\") pod \"simulator-w0-bc102e-kserve-798dd5948b-7tnxt\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:10.105540 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.105522 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-home\") pod \"simulator-w0-bc102e-kserve-798dd5948b-7tnxt\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:10.106009 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.105992 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-tmp-dir\") pod \"simulator-w0-bc102e-kserve-798dd5948b-7tnxt\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:10.106761 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.106741 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-tls-certs\") pod \"simulator-w0-bc102e-kserve-798dd5948b-7tnxt\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:10.106960 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.104856 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-model-cache\") pod \"simulator-w0-bc102e-kserve-798dd5948b-7tnxt\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:10.107071 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.106342 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-tmp-dir\") pod \"simulator-w0-bc102e-kserve-798dd5948b-7tnxt\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:10.107169 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.105937 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-home\") pod \"simulator-w0-bc102e-kserve-798dd5948b-7tnxt\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:10.112216 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.112186 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-dshm\") pod \"simulator-w0-bc102e-kserve-798dd5948b-7tnxt\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:10.112359 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.112326 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-tls-certs\") pod \"simulator-w0-bc102e-kserve-798dd5948b-7tnxt\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:10.115133 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.115106 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqrjg\" (UniqueName: \"kubernetes.io/projected/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-kube-api-access-pqrjg\") pod \"simulator-w0-bc102e-kserve-798dd5948b-7tnxt\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:10.248635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.248551 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:10.525259 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.525233 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt"] Oct 30 20:48:10.527982 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:48:10.527951 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa47948a_4865_45d9_a687_fc7fbcc3a6a8.slice/crio-4c098752cbbf792ee4bdd76002e4951152b2a2e1a2dd5482f7b10b9b6560eb02 WatchSource:0}: Error finding container 4c098752cbbf792ee4bdd76002e4951152b2a2e1a2dd5482f7b10b9b6560eb02: Status 404 returned error can't find the container with id 4c098752cbbf792ee4bdd76002e4951152b2a2e1a2dd5482f7b10b9b6560eb02 Oct 30 20:48:10.873901 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.873865 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" event={"ID":"fa47948a-4865-45d9-a687-fc7fbcc3a6a8","Type":"ContainerStarted","Data":"e366d5861fd5412c5af70e1fc81901b6afde74226dc13beda28a50f5d8e3ae20"} Oct 30 20:48:10.874046 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.873910 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" event={"ID":"fa47948a-4865-45d9-a687-fc7fbcc3a6a8","Type":"ContainerStarted","Data":"4c098752cbbf792ee4bdd76002e4951152b2a2e1a2dd5482f7b10b9b6560eb02"} Oct 30 20:48:10.890964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:10.890922 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podStartSLOduration=1.890907369 podStartE2EDuration="1.890907369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:10.890076818 +0000 UTC m=+596.069973326" watchObservedRunningTime="2026-10-30 20:48:10.890907369 +0000 UTC m=+596.070803877" Oct 30 20:48:11.248865 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:11.248830 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:13.352703 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.352211 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp"] Oct 30 20:48:13.358150 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.358123 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:13.367957 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.367928 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp"] Oct 30 20:48:13.368099 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.367958 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f9a819\"/\"openshift-service-ca.crt\"" Oct 30 20:48:13.368099 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.368008 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-f9a819\"/\"kube-root-ca.crt\"" Oct 30 20:48:13.368099 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.368008 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f9a819\"/\"premium-w3-f9a819-kserve-self-signed-certs\"" Oct 30 20:48:13.368099 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.368065 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f9a819\"/\"default-dockercfg-mvzcp\"" Oct 30 20:48:13.448229 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.448196 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk4nn\" (UniqueName: \"kubernetes.io/projected/dc3af69c-3d4e-4aab-98af-9082f94ace11-kube-api-access-rk4nn\") pod \"premium-w3-f9a819-kserve-7b56fdfd69-th6hp\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:13.448396 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.448243 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-dshm\") pod \"premium-w3-f9a819-kserve-7b56fdfd69-th6hp\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:13.448396 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.448311 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-home\") pod \"premium-w3-f9a819-kserve-7b56fdfd69-th6hp\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:13.448396 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.448336 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3af69c-3d4e-4aab-98af-9082f94ace11-tls-certs\") pod \"premium-w3-f9a819-kserve-7b56fdfd69-th6hp\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:13.448396 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.448375 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-tmp-dir\") pod \"premium-w3-f9a819-kserve-7b56fdfd69-th6hp\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:13.448586 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.448492 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-model-cache\") pod \"premium-w3-f9a819-kserve-7b56fdfd69-th6hp\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:13.549625 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.549587 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk4nn\" (UniqueName: \"kubernetes.io/projected/dc3af69c-3d4e-4aab-98af-9082f94ace11-kube-api-access-rk4nn\") pod \"premium-w3-f9a819-kserve-7b56fdfd69-th6hp\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:13.549847 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.549647 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-dshm\") pod \"premium-w3-f9a819-kserve-7b56fdfd69-th6hp\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:13.549847 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.549684 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-home\") pod \"premium-w3-f9a819-kserve-7b56fdfd69-th6hp\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:13.549847 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.549709 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3af69c-3d4e-4aab-98af-9082f94ace11-tls-certs\") pod \"premium-w3-f9a819-kserve-7b56fdfd69-th6hp\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:13.549847 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.549783 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-tmp-dir\") pod \"premium-w3-f9a819-kserve-7b56fdfd69-th6hp\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:13.549847 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.549824 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-model-cache\") pod \"premium-w3-f9a819-kserve-7b56fdfd69-th6hp\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:13.550190 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.550168 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-home\") pod \"premium-w3-f9a819-kserve-7b56fdfd69-th6hp\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:13.550237 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.550203 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-tmp-dir\") pod \"premium-w3-f9a819-kserve-7b56fdfd69-th6hp\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:13.550296 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.550237 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-model-cache\") pod \"premium-w3-f9a819-kserve-7b56fdfd69-th6hp\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:13.552177 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.552154 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-dshm\") pod \"premium-w3-f9a819-kserve-7b56fdfd69-th6hp\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:13.552439 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.552422 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3af69c-3d4e-4aab-98af-9082f94ace11-tls-certs\") pod \"premium-w3-f9a819-kserve-7b56fdfd69-th6hp\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:13.558554 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.558532 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk4nn\" (UniqueName: \"kubernetes.io/projected/dc3af69c-3d4e-4aab-98af-9082f94ace11-kube-api-access-rk4nn\") pod \"premium-w3-f9a819-kserve-7b56fdfd69-th6hp\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:13.670451 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.670366 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:13.944667 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:13.944639 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp"] Oct 30 20:48:13.946791 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:48:13.946751 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc3af69c_3d4e_4aab_98af_9082f94ace11.slice/crio-5e98265dd18080f2a490813034f59aea9addc8d17a28618e8c73c0206e0cab2a WatchSource:0}: Error finding container 5e98265dd18080f2a490813034f59aea9addc8d17a28618e8c73c0206e0cab2a: Status 404 returned error can't find the container with id 5e98265dd18080f2a490813034f59aea9addc8d17a28618e8c73c0206e0cab2a Oct 30 20:48:14.898355 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:14.898324 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" event={"ID":"dc3af69c-3d4e-4aab-98af-9082f94ace11","Type":"ContainerStarted","Data":"867cc8fba123252ecc85d54799cf002bdead40cab09a401a42cc72fcd045e204"} Oct 30 20:48:14.898355 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:14.898360 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" event={"ID":"dc3af69c-3d4e-4aab-98af-9082f94ace11","Type":"ContainerStarted","Data":"5e98265dd18080f2a490813034f59aea9addc8d17a28618e8c73c0206e0cab2a"} Oct 30 20:48:14.918607 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:14.918561 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podStartSLOduration=1.918546406 podStartE2EDuration="1.918546406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:14.916684392 +0000 UTC m=+600.096580900" watchObservedRunningTime="2026-10-30 20:48:14.918546406 +0000 UTC m=+600.098442913" Oct 30 20:48:14.986042 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:14.986011 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 20:48:14.988149 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:14.988123 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 20:48:15.671408 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:15.671379 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:20.248854 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:20.248815 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:20.261231 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:20.261205 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:20.940945 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:20.940918 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:48:22.509681 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:48:22.509635 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:48:23.670918 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:23.670886 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:23.683265 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:23.683241 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:23.955662 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:23.955632 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:48:41.162633 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.162592 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g"] Oct 30 20:48:41.168262 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.168237 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:41.179823 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.179759 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8e4acb\"/\"openshift-service-ca.crt\"" Oct 30 20:48:41.180048 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.180010 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-8e4acb\"/\"kube-root-ca.crt\"" Oct 30 20:48:41.180912 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.180890 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g"] Oct 30 20:48:41.184106 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.184085 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8e4acb\"/\"premium-w2-8e4acb-kserve-self-signed-certs\"" Oct 30 20:48:41.184200 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.184177 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-8e4acb\"/\"default-dockercfg-5f97g\"" Oct 30 20:48:41.270938 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.270897 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shtp5\" (UniqueName: \"kubernetes.io/projected/3111f52b-8837-47a7-9dc2-1da7d0574d1f-kube-api-access-shtp5\") pod \"premium-w2-8e4acb-kserve-7b5b878587-ptp4g\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:41.270938 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.270938 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-tmp-dir\") pod \"premium-w2-8e4acb-kserve-7b5b878587-ptp4g\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:41.271199 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.271013 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-dshm\") pod \"premium-w2-8e4acb-kserve-7b5b878587-ptp4g\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:41.271199 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.271074 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-model-cache\") pod \"premium-w2-8e4acb-kserve-7b5b878587-ptp4g\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:41.271199 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.271095 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3111f52b-8837-47a7-9dc2-1da7d0574d1f-tls-certs\") pod \"premium-w2-8e4acb-kserve-7b5b878587-ptp4g\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:41.271199 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.271190 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-home\") pod \"premium-w2-8e4acb-kserve-7b5b878587-ptp4g\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:41.371708 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.371666 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-home\") pod \"premium-w2-8e4acb-kserve-7b5b878587-ptp4g\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:41.371924 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.371726 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shtp5\" (UniqueName: \"kubernetes.io/projected/3111f52b-8837-47a7-9dc2-1da7d0574d1f-kube-api-access-shtp5\") pod \"premium-w2-8e4acb-kserve-7b5b878587-ptp4g\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:41.371924 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.371777 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-tmp-dir\") pod \"premium-w2-8e4acb-kserve-7b5b878587-ptp4g\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:41.371924 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.371815 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-dshm\") pod \"premium-w2-8e4acb-kserve-7b5b878587-ptp4g\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:41.371924 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.371915 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-model-cache\") pod \"premium-w2-8e4acb-kserve-7b5b878587-ptp4g\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:41.372196 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.372152 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-home\") pod \"premium-w2-8e4acb-kserve-7b5b878587-ptp4g\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:41.372196 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.372165 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3111f52b-8837-47a7-9dc2-1da7d0574d1f-tls-certs\") pod \"premium-w2-8e4acb-kserve-7b5b878587-ptp4g\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:41.372310 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.372193 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-tmp-dir\") pod \"premium-w2-8e4acb-kserve-7b5b878587-ptp4g\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:41.372310 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.372215 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-model-cache\") pod \"premium-w2-8e4acb-kserve-7b5b878587-ptp4g\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:41.374482 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.374450 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-dshm\") pod \"premium-w2-8e4acb-kserve-7b5b878587-ptp4g\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:41.374881 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.374857 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3111f52b-8837-47a7-9dc2-1da7d0574d1f-tls-certs\") pod \"premium-w2-8e4acb-kserve-7b5b878587-ptp4g\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:41.391044 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.391008 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shtp5\" (UniqueName: \"kubernetes.io/projected/3111f52b-8837-47a7-9dc2-1da7d0574d1f-kube-api-access-shtp5\") pod \"premium-w2-8e4acb-kserve-7b5b878587-ptp4g\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:41.482360 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.482319 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:41.774697 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:41.774662 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g"] Oct 30 20:48:42.065850 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:42.065546 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" event={"ID":"3111f52b-8837-47a7-9dc2-1da7d0574d1f","Type":"ContainerStarted","Data":"11c80ebcee5d12505ed3675e680d3758ae82d311e50d41411fac77ac940a1713"} Oct 30 20:48:42.065850 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:42.065605 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" event={"ID":"3111f52b-8837-47a7-9dc2-1da7d0574d1f","Type":"ContainerStarted","Data":"9babff7f575a77a23d2685c901628faeb98478d71752435d4afe90e544b630ac"} Oct 30 20:48:42.092449 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:42.092392 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podStartSLOduration=1.092377571 podStartE2EDuration="1.092377571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:42.089598518 +0000 UTC m=+627.269495019" watchObservedRunningTime="2026-10-30 20:48:42.092377571 +0000 UTC m=+627.272274112" Oct 30 20:48:42.483120 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:42.483083 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:51.482550 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:51.482508 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:51.495514 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:51.495484 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:52.131856 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:52.131826 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:48:52.510143 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:48:52.510100 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:48:53.904880 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:53.904847 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m"] Oct 30 20:48:53.908757 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:53.908739 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:53.913628 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:53.913604 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-317a33-openshift-default-dockercfg-9qjvh\"" Oct 30 20:48:53.923927 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:53.923903 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m"] Oct 30 20:48:54.032677 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.032637 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.032897 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.032691 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.032897 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.032843 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm4xv\" (UniqueName: \"kubernetes.io/projected/16e788c7-69cc-48cb-959a-62c4763c2698-kube-api-access-rm4xv\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.033030 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.033000 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.033087 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.033052 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.033159 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.033142 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e788c7-69cc-48cb-959a-62c4763c2698-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.033223 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.033207 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.033284 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.033243 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e788c7-69cc-48cb-959a-62c4763c2698-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.033339 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.033296 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e788c7-69cc-48cb-959a-62c4763c2698-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.134694 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.134662 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.134892 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.134706 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.134892 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.134725 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e788c7-69cc-48cb-959a-62c4763c2698-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.134892 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.134756 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.134892 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.134789 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e788c7-69cc-48cb-959a-62c4763c2698-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.134892 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.134813 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e788c7-69cc-48cb-959a-62c4763c2698-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.134892 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.134844 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.134892 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.134872 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.135308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.134910 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm4xv\" (UniqueName: \"kubernetes.io/projected/16e788c7-69cc-48cb-959a-62c4763c2698-kube-api-access-rm4xv\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.135308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.135293 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.135410 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.135293 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.135474 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.135443 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.135531 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.135515 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.135738 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.135718 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e788c7-69cc-48cb-959a-62c4763c2698-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.136954 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.136932 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.137222 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.137205 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e788c7-69cc-48cb-959a-62c4763c2698-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.146214 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.146195 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e788c7-69cc-48cb-959a-62c4763c2698-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.146324 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.146308 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm4xv\" (UniqueName: \"kubernetes.io/projected/16e788c7-69cc-48cb-959a-62c4763c2698-kube-api-access-rm4xv\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.222640 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.222608 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:54.503829 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.503800 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m"] Oct 30 20:48:54.504589 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:48:54.504562 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e788c7_69cc_48cb_959a_62c4763c2698.slice/crio-fad9eb3a9a6e5c655857f7f34c42af038f0595d06a1eef8b24b2cf7cde2dd472 WatchSource:0}: Error finding container fad9eb3a9a6e5c655857f7f34c42af038f0595d06a1eef8b24b2cf7cde2dd472: Status 404 returned error can't find the container with id fad9eb3a9a6e5c655857f7f34c42af038f0595d06a1eef8b24b2cf7cde2dd472 Oct 30 20:48:54.506996 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.506957 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:48:54.507095 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.507044 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:48:54.507159 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:54.507114 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:48:55.138837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:55.138800 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" event={"ID":"16e788c7-69cc-48cb-959a-62c4763c2698","Type":"ContainerStarted","Data":"97484869dee74327eaf742fa17348a1abddd4ca77456e3d2017caa1046a662fd"} Oct 30 20:48:55.139265 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:55.138868 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" event={"ID":"16e788c7-69cc-48cb-959a-62c4763c2698","Type":"ContainerStarted","Data":"fad9eb3a9a6e5c655857f7f34c42af038f0595d06a1eef8b24b2cf7cde2dd472"} Oct 30 20:48:55.170671 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:55.170603 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" podStartSLOduration=2.170570678 podStartE2EDuration="2.170570678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:48:55.169378607 +0000 UTC m=+640.349275115" watchObservedRunningTime="2026-10-30 20:48:55.170570678 +0000 UTC m=+640.350467185" Oct 30 20:48:55.225495 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:55.225462 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:48:55.229918 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:55.229874 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:48:55.230096 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:55.229960 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" podUID="16e788c7-69cc-48cb-959a-62c4763c2698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 20:48:56.226793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:56.226195 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:48:56.226793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:56.226275 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" podUID="16e788c7-69cc-48cb-959a-62c4763c2698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 20:48:57.228789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:57.228169 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:48:57.228789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:57.228233 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" podUID="16e788c7-69cc-48cb-959a-62c4763c2698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 20:48:58.224786 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:58.224184 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:48:58.224786 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:58.224243 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" podUID="16e788c7-69cc-48cb-959a-62c4763c2698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 20:48:59.225925 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:59.225888 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:48:59.226494 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:48:59.226462 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" podUID="16e788c7-69cc-48cb-959a-62c4763c2698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 20:49:00.225791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:00.225163 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 20:49:00.225791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:00.225226 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" podUID="16e788c7-69cc-48cb-959a-62c4763c2698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 20:49:01.439312 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:01.439276 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:49:01.439854 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:01.439713 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:49:01.441561 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:01.441538 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:49:22.514861 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:49:22.514745 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:49:26.380228 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.380198 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf"] Oct 30 20:49:26.384079 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.384058 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" Oct 30 20:49:26.396456 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.395057 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 20:49:26.396456 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.395145 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-28159f2b53e5\"" Oct 30 20:49:26.396456 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.395228 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 20:49:26.396456 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.395253 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 20:49:26.396456 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.395283 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 20:49:26.396456 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.395398 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-qn9bd\"" Oct 30 20:49:26.396868 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.396854 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf"] Oct 30 20:49:26.397888 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.397862 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7d1c765a6192\"" Oct 30 20:49:26.452949 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.452910 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6d9b62c-2a59-4223-805b-33263b17d846-provider-credentials\") pod \"payload-processing-external-model-d65b6b7bb-k98hf\" (UID: \"a6d9b62c-2a59-4223-805b-33263b17d846\") " pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" Oct 30 20:49:26.453137 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.452979 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d9b62c-2a59-4223-805b-33263b17d846-plugins-config-volume\") pod \"payload-processing-external-model-d65b6b7bb-k98hf\" (UID: \"a6d9b62c-2a59-4223-805b-33263b17d846\") " pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" Oct 30 20:49:26.453137 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.453099 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r77tz\" (UniqueName: \"kubernetes.io/projected/a6d9b62c-2a59-4223-805b-33263b17d846-kube-api-access-r77tz\") pod \"payload-processing-external-model-d65b6b7bb-k98hf\" (UID: \"a6d9b62c-2a59-4223-805b-33263b17d846\") " pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" Oct 30 20:49:26.453262 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.453175 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6d9b62c-2a59-4223-805b-33263b17d846-routing-overlay-volume\") pod \"payload-processing-external-model-d65b6b7bb-k98hf\" (UID: \"a6d9b62c-2a59-4223-805b-33263b17d846\") " pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" Oct 30 20:49:26.554817 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.554759 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6d9b62c-2a59-4223-805b-33263b17d846-routing-overlay-volume\") pod \"payload-processing-external-model-d65b6b7bb-k98hf\" (UID: \"a6d9b62c-2a59-4223-805b-33263b17d846\") " pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" Oct 30 20:49:26.554992 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.554877 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6d9b62c-2a59-4223-805b-33263b17d846-provider-credentials\") pod \"payload-processing-external-model-d65b6b7bb-k98hf\" (UID: \"a6d9b62c-2a59-4223-805b-33263b17d846\") " pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" Oct 30 20:49:26.554992 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.554911 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d9b62c-2a59-4223-805b-33263b17d846-plugins-config-volume\") pod \"payload-processing-external-model-d65b6b7bb-k98hf\" (UID: \"a6d9b62c-2a59-4223-805b-33263b17d846\") " pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" Oct 30 20:49:26.554992 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.554950 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r77tz\" (UniqueName: \"kubernetes.io/projected/a6d9b62c-2a59-4223-805b-33263b17d846-kube-api-access-r77tz\") pod \"payload-processing-external-model-d65b6b7bb-k98hf\" (UID: \"a6d9b62c-2a59-4223-805b-33263b17d846\") " pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" Oct 30 20:49:26.555609 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.555566 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6d9b62c-2a59-4223-805b-33263b17d846-routing-overlay-volume\") pod \"payload-processing-external-model-d65b6b7bb-k98hf\" (UID: \"a6d9b62c-2a59-4223-805b-33263b17d846\") " pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" Oct 30 20:49:26.555797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.555704 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d9b62c-2a59-4223-805b-33263b17d846-plugins-config-volume\") pod \"payload-processing-external-model-d65b6b7bb-k98hf\" (UID: \"a6d9b62c-2a59-4223-805b-33263b17d846\") " pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" Oct 30 20:49:26.558002 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.557975 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6d9b62c-2a59-4223-805b-33263b17d846-provider-credentials\") pod \"payload-processing-external-model-d65b6b7bb-k98hf\" (UID: \"a6d9b62c-2a59-4223-805b-33263b17d846\") " pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" Oct 30 20:49:26.565000 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.564969 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r77tz\" (UniqueName: \"kubernetes.io/projected/a6d9b62c-2a59-4223-805b-33263b17d846-kube-api-access-r77tz\") pod \"payload-processing-external-model-d65b6b7bb-k98hf\" (UID: \"a6d9b62c-2a59-4223-805b-33263b17d846\") " pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" Oct 30 20:49:26.704091 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.704059 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" Oct 30 20:49:26.986681 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:26.986649 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf"] Oct 30 20:49:26.987634 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:49:26.987602 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d9b62c_2a59_4223_805b_33263b17d846.slice/crio-20a9d0e31fe38f611e84d7fc47d7e39d8c25a1875e10fa8c8d657e5398da4bd7 WatchSource:0}: Error finding container 20a9d0e31fe38f611e84d7fc47d7e39d8c25a1875e10fa8c8d657e5398da4bd7: Status 404 returned error can't find the container with id 20a9d0e31fe38f611e84d7fc47d7e39d8c25a1875e10fa8c8d657e5398da4bd7 Oct 30 20:49:27.341273 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.341178 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" event={"ID":"a6d9b62c-2a59-4223-805b-33263b17d846","Type":"ContainerStarted","Data":"48197571f659be6b5eec296aa4d171170bf82d136f9cad5950e7362ab0efe53f"} Oct 30 20:49:27.341273 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.341243 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" Oct 30 20:49:27.341273 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.341266 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" event={"ID":"a6d9b62c-2a59-4223-805b-33263b17d846","Type":"ContainerStarted","Data":"20a9d0e31fe38f611e84d7fc47d7e39d8c25a1875e10fa8c8d657e5398da4bd7"} Oct 30 20:49:27.364825 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.364739 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" podStartSLOduration=1.3647241 podStartE2EDuration="1.3647241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:27.361521697 +0000 UTC m=+672.541418205" watchObservedRunningTime="2026-10-30 20:49:27.3647241 +0000 UTC m=+672.544620606" Oct 30 20:49:27.775738 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.775692 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25"] Oct 30 20:49:27.779912 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.779885 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.785599 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.785575 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-8e4acb-openshift-default-dockercfg-w4qvm\"" Oct 30 20:49:27.794705 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.794681 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25"] Oct 30 20:49:27.876791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.875522 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.876791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.875575 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.876791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.875664 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.876791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.875713 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.876791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.875743 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.876791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.875809 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.876791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.875861 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4z2s\" (UniqueName: \"kubernetes.io/projected/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-kube-api-access-n4z2s\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.876791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.875902 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.876791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.875961 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.980787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.977380 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.980787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.977434 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.980787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.977481 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.980787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.977534 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4z2s\" (UniqueName: \"kubernetes.io/projected/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-kube-api-access-n4z2s\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.980787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.977577 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.980787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.977633 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.980787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.977679 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.980787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.977706 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.980787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.977812 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.980787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.978287 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.986756 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.983335 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.987230 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.987207 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.989027 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.989003 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:27.989397 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:27.989375 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:28.002526 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:28.002488 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:28.010803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:28.006742 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:28.013845 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:28.013428 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4z2s\" (UniqueName: \"kubernetes.io/projected/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-kube-api-access-n4z2s\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:28.135683 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:28.135596 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:28.399963 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:28.399874 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:28.810859 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:28.810822 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25"] Oct 30 20:49:28.814611 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:49:28.814577 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2c7b5c_94f9_45b8_ad43_de9a1f5110f2.slice/crio-68a17bc2b3f4aaef72eee5adc30b8ccbb28b83b42a48980be3c62776fab823c3 WatchSource:0}: Error finding container 68a17bc2b3f4aaef72eee5adc30b8ccbb28b83b42a48980be3c62776fab823c3: Status 404 returned error can't find the container with id 68a17bc2b3f4aaef72eee5adc30b8ccbb28b83b42a48980be3c62776fab823c3 Oct 30 20:49:28.817501 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:28.817464 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:28.817612 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:28.817545 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:28.817612 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:28.817588 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:29.377802 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:29.377747 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" event={"ID":"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2","Type":"ContainerStarted","Data":"dddd23d9dc62bf73bd08590b1ad386ef265d3286438e556da72dd6c5880c4dd5"} Oct 30 20:49:29.377977 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:29.377822 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" event={"ID":"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2","Type":"ContainerStarted","Data":"68a17bc2b3f4aaef72eee5adc30b8ccbb28b83b42a48980be3c62776fab823c3"} Oct 30 20:49:29.401289 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:29.401264 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:29.402915 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:29.402893 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:49:29.403025 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:29.402933 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" podUID="ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 20:49:29.410212 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:29.410151 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" podStartSLOduration=2.410133584 podStartE2EDuration="2.410133584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:29.407668508 +0000 UTC m=+674.587565016" watchObservedRunningTime="2026-10-30 20:49:29.410133584 +0000 UTC m=+674.590030094" Oct 30 20:49:30.401486 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:30.401454 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:49:30.401992 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:30.401508 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" podUID="ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 20:49:31.400527 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:31.400485 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:49:31.400749 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:31.400542 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" podUID="ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 20:49:32.401300 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:32.401263 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:49:32.401795 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:32.401359 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" podUID="ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 20:49:33.400702 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:33.400665 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:49:33.400953 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:33.400736 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" podUID="ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 20:49:34.400537 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:34.400504 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:49:34.401028 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:34.400557 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" podUID="ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 20:49:35.400930 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:35.400890 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 20:49:35.401333 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:35.400974 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" podUID="ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 20:49:36.405722 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:36.405695 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:36.406260 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:36.406225 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:36.407387 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:36.407367 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 20:49:37.241747 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.241707 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc"] Oct 30 20:49:37.246002 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.245973 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.260209 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.260179 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc"] Oct 30 20:49:37.311184 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.311151 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0226107b-80ef-4c6e-888c-f89f73108c20-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.311184 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.311186 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.311438 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.311218 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.311438 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.311238 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.311559 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.311444 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvrkf\" (UniqueName: \"kubernetes.io/projected/0226107b-80ef-4c6e-888c-f89f73108c20-kube-api-access-bvrkf\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.311559 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.311498 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0226107b-80ef-4c6e-888c-f89f73108c20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.311664 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.311590 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.311664 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.311651 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0226107b-80ef-4c6e-888c-f89f73108c20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.311806 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.311678 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.412938 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.412897 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvrkf\" (UniqueName: \"kubernetes.io/projected/0226107b-80ef-4c6e-888c-f89f73108c20-kube-api-access-bvrkf\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.413441 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.412956 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0226107b-80ef-4c6e-888c-f89f73108c20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.413441 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.413006 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.413441 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.413052 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0226107b-80ef-4c6e-888c-f89f73108c20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.413441 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.413082 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.413441 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.413174 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0226107b-80ef-4c6e-888c-f89f73108c20-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.413441 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.413205 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.413441 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.413259 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.413441 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.413293 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.413941 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.413485 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.413941 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.413688 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.413941 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.413741 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.414113 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.413976 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.414113 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.414027 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0226107b-80ef-4c6e-888c-f89f73108c20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.416335 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.416304 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.416885 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.416865 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0226107b-80ef-4c6e-888c-f89f73108c20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.423306 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.423281 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0226107b-80ef-4c6e-888c-f89f73108c20-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.423947 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.423926 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvrkf\" (UniqueName: \"kubernetes.io/projected/0226107b-80ef-4c6e-888c-f89f73108c20-kube-api-access-bvrkf\") pod \"maas-default-gateway-openshift-default-77f556d647-mfvmc\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.563355 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.563244 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:37.867478 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.867377 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc"] Oct 30 20:49:37.872984 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:49:37.872951 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0226107b_80ef_4c6e_888c_f89f73108c20.slice/crio-c7f8a1b05245e229656320ea7e7bb2633f0f8d6cfde4b68092a2d5c76d2dd64b WatchSource:0}: Error finding container c7f8a1b05245e229656320ea7e7bb2633f0f8d6cfde4b68092a2d5c76d2dd64b: Status 404 returned error can't find the container with id c7f8a1b05245e229656320ea7e7bb2633f0f8d6cfde4b68092a2d5c76d2dd64b Oct 30 20:49:37.876149 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.876104 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:37.876254 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.876190 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:37.876254 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:37.876231 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:38.375530 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:38.375495 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" Oct 30 20:49:38.458082 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:38.458050 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" event={"ID":"0226107b-80ef-4c6e-888c-f89f73108c20","Type":"ContainerStarted","Data":"072192f1b8a41c1fa6a026efc64d597a75285d036e5aac6af8f32ac70860f210"} Oct 30 20:49:38.458486 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:38.458108 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" event={"ID":"0226107b-80ef-4c6e-888c-f89f73108c20","Type":"ContainerStarted","Data":"c7f8a1b05245e229656320ea7e7bb2633f0f8d6cfde4b68092a2d5c76d2dd64b"} Oct 30 20:49:38.483795 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:38.482198 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" podStartSLOduration=1.482178508 podStartE2EDuration="1.482178508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:38.48122189 +0000 UTC m=+683.661118397" watchObservedRunningTime="2026-10-30 20:49:38.482178508 +0000 UTC m=+683.662075019" Oct 30 20:49:38.564080 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:38.564044 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:38.565911 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:38.565887 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-mfvmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 20:49:38.566023 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:38.565940 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" podUID="0226107b-80ef-4c6e-888c-f89f73108c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 20:49:39.564108 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:39.564072 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-mfvmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 20:49:39.564505 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:39.564129 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" podUID="0226107b-80ef-4c6e-888c-f89f73108c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 20:49:40.563850 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:40.563814 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-mfvmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 20:49:40.564022 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:40.563880 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" podUID="0226107b-80ef-4c6e-888c-f89f73108c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 20:49:41.564498 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:41.564456 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-mfvmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 20:49:41.565015 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:41.564522 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" podUID="0226107b-80ef-4c6e-888c-f89f73108c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 20:49:42.564080 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:42.564027 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-mfvmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 20:49:42.564296 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:42.564108 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" podUID="0226107b-80ef-4c6e-888c-f89f73108c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 20:49:43.563972 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:43.563939 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-mfvmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 20:49:43.564362 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:43.563991 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" podUID="0226107b-80ef-4c6e-888c-f89f73108c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 20:49:44.564504 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:44.564461 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-mfvmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 20:49:44.564882 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:44.564518 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" podUID="0226107b-80ef-4c6e-888c-f89f73108c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 20:49:46.336254 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.336217 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:46.336732 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.336710 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:46.337680 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.337656 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 20:49:46.707272 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.707242 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd"] Oct 30 20:49:46.711059 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.711037 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.729861 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.729831 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd"] Oct 30 20:49:46.844585 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.844550 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.844817 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.844610 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932dfb46-276c-4d55-9800-f1cbaf96360d-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.844817 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.844739 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.844954 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.844843 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.844954 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.844919 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.845028 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.844991 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.845028 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.845022 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.845135 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.845119 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpg9w\" (UniqueName: \"kubernetes.io/projected/932dfb46-276c-4d55-9800-f1cbaf96360d-kube-api-access-vpg9w\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.845170 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.845148 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.946278 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.946238 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932dfb46-276c-4d55-9800-f1cbaf96360d-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.946444 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.946303 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.946444 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.946333 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.946444 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.946355 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.946444 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.946395 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.946444 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.946427 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.946710 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.946465 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpg9w\" (UniqueName: \"kubernetes.io/projected/932dfb46-276c-4d55-9800-f1cbaf96360d-kube-api-access-vpg9w\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.946710 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.946495 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.946710 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.946568 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.947037 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.946848 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.947037 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.946875 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.947037 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.946928 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.947202 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.947100 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.947202 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.947155 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932dfb46-276c-4d55-9800-f1cbaf96360d-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.949103 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.949074 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.949365 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.949344 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.955150 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.955127 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:46.955382 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:46.955362 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpg9w\" (UniqueName: \"kubernetes.io/projected/932dfb46-276c-4d55-9800-f1cbaf96360d-kube-api-access-vpg9w\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:47.035250 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:47.035159 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:47.319289 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:47.319259 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd"] Oct 30 20:49:47.320813 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:49:47.320785 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932dfb46_276c_4d55_9800_f1cbaf96360d.slice/crio-87fbf9340a4a7401810cd185143825fe0b3c5d0d16b3459b0b728e2fe4aa9ad9 WatchSource:0}: Error finding container 87fbf9340a4a7401810cd185143825fe0b3c5d0d16b3459b0b728e2fe4aa9ad9: Status 404 returned error can't find the container with id 87fbf9340a4a7401810cd185143825fe0b3c5d0d16b3459b0b728e2fe4aa9ad9 Oct 30 20:49:47.323232 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:47.323201 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:47.323311 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:47.323265 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:47.323311 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:47.323293 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:47.514423 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:47.514253 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" event={"ID":"932dfb46-276c-4d55-9800-f1cbaf96360d","Type":"ContainerStarted","Data":"cd3adba4e5182237fcf71de4636acd6c3e4f9f919ffef0e56881a5df452db327"} Oct 30 20:49:47.514423 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:47.514348 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" event={"ID":"932dfb46-276c-4d55-9800-f1cbaf96360d","Type":"ContainerStarted","Data":"87fbf9340a4a7401810cd185143825fe0b3c5d0d16b3459b0b728e2fe4aa9ad9"} Oct 30 20:49:47.537397 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:47.537338 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" podStartSLOduration=1.5373238329999999 podStartE2EDuration="1.537323833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:47.534071049 +0000 UTC m=+692.713967562" watchObservedRunningTime="2026-10-30 20:49:47.537323833 +0000 UTC m=+692.717220340" Oct 30 20:49:48.036081 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:48.036050 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:48.037272 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:48.037250 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 20:49:48.037348 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:48.037299 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" podUID="932dfb46-276c-4d55-9800-f1cbaf96360d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 20:49:49.036198 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:49.036161 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 20:49:49.036591 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:49.036222 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" podUID="932dfb46-276c-4d55-9800-f1cbaf96360d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 20:49:50.039855 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:50.037228 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 20:49:50.039855 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:50.037289 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" podUID="932dfb46-276c-4d55-9800-f1cbaf96360d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 20:49:51.035683 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:51.035650 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 20:49:51.035967 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:51.035937 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" podUID="932dfb46-276c-4d55-9800-f1cbaf96360d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 20:49:52.035548 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.035506 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 20:49:52.036080 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.035581 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" podUID="932dfb46-276c-4d55-9800-f1cbaf96360d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 20:49:52.347452 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.347368 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9"] Oct 30 20:49:52.351717 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.351695 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.381971 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.381932 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9"] Oct 30 20:49:52.446025 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.445988 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6fc6562-45cb-47db-98ab-415e184c71b3-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.446205 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.446033 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6fc6562-45cb-47db-98ab-415e184c71b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.446205 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.446098 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6fc6562-45cb-47db-98ab-415e184c71b3-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.446205 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.446132 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vswq4\" (UniqueName: \"kubernetes.io/projected/a6fc6562-45cb-47db-98ab-415e184c71b3-kube-api-access-vswq4\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.446319 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.446272 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6fc6562-45cb-47db-98ab-415e184c71b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.446374 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.446318 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fc6562-45cb-47db-98ab-415e184c71b3-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.446442 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.446424 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6fc6562-45cb-47db-98ab-415e184c71b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.446509 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.446496 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fc6562-45cb-47db-98ab-415e184c71b3-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.446563 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.446523 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6fc6562-45cb-47db-98ab-415e184c71b3-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.535815 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:49:52.535758 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:49:52.547488 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.547460 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6fc6562-45cb-47db-98ab-415e184c71b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.547632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.547494 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fc6562-45cb-47db-98ab-415e184c71b3-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.547632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.547543 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6fc6562-45cb-47db-98ab-415e184c71b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.547632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.547587 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fc6562-45cb-47db-98ab-415e184c71b3-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.547632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.547610 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6fc6562-45cb-47db-98ab-415e184c71b3-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.547884 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.547718 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6fc6562-45cb-47db-98ab-415e184c71b3-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.547884 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.547745 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6fc6562-45cb-47db-98ab-415e184c71b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.547884 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.547789 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6fc6562-45cb-47db-98ab-415e184c71b3-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.547884 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.547823 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vswq4\" (UniqueName: \"kubernetes.io/projected/a6fc6562-45cb-47db-98ab-415e184c71b3-kube-api-access-vswq4\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.548099 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.548021 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fc6562-45cb-47db-98ab-415e184c71b3-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.548169 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.548090 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6fc6562-45cb-47db-98ab-415e184c71b3-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.548321 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.548298 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6fc6562-45cb-47db-98ab-415e184c71b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.548457 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.548432 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6fc6562-45cb-47db-98ab-415e184c71b3-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.548457 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.548386 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6fc6562-45cb-47db-98ab-415e184c71b3-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.550025 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.550006 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6fc6562-45cb-47db-98ab-415e184c71b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.550253 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.550235 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6fc6562-45cb-47db-98ab-415e184c71b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.557350 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.557328 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6fc6562-45cb-47db-98ab-415e184c71b3-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.557567 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.557548 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vswq4\" (UniqueName: \"kubernetes.io/projected/a6fc6562-45cb-47db-98ab-415e184c71b3-kube-api-access-vswq4\") pod \"maas-default-gateway-openshift-default-77f556d647-sb5c9\" (UID: \"a6fc6562-45cb-47db-98ab-415e184c71b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.667137 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.667056 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:52.986201 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.986169 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9"] Oct 30 20:49:52.986851 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:49:52.986824 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6fc6562_45cb_47db_98ab_415e184c71b3.slice/crio-e1afa7b3d4bb0828ae1e366d7d35d93bf34231239e0cd67b6d40fa655d700a84 WatchSource:0}: Error finding container e1afa7b3d4bb0828ae1e366d7d35d93bf34231239e0cd67b6d40fa655d700a84: Status 404 returned error can't find the container with id e1afa7b3d4bb0828ae1e366d7d35d93bf34231239e0cd67b6d40fa655d700a84 Oct 30 20:49:52.989342 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.989309 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:52.989435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.989383 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:52.989435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:52.989413 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:53.036114 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:53.036084 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 20:49:53.036475 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:53.036132 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" podUID="932dfb46-276c-4d55-9800-f1cbaf96360d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 20:49:53.588492 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:53.588447 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" event={"ID":"a6fc6562-45cb-47db-98ab-415e184c71b3","Type":"ContainerStarted","Data":"ce1babc8dc9d91c7c8a1093add0ff599979994f11899bfe844eb50af43aba393"} Oct 30 20:49:53.588650 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:53.588519 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" event={"ID":"a6fc6562-45cb-47db-98ab-415e184c71b3","Type":"ContainerStarted","Data":"e1afa7b3d4bb0828ae1e366d7d35d93bf34231239e0cd67b6d40fa655d700a84"} Oct 30 20:49:53.623651 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:53.623569 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" podStartSLOduration=1.623547309 podStartE2EDuration="1.623547309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:53.618253442 +0000 UTC m=+698.798149949" watchObservedRunningTime="2026-10-30 20:49:53.623547309 +0000 UTC m=+698.803443817" Oct 30 20:49:53.667781 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:53.667733 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:49:53.996716 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:53.996673 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24"] Oct 30 20:49:54.001170 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.001142 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.041684 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.041643 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 20:49:54.042218 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.041705 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24"] Oct 30 20:49:54.042218 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.042112 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" podUID="932dfb46-276c-4d55-9800-f1cbaf96360d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 20:49:54.202462 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.202416 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzc27\" (UniqueName: \"kubernetes.io/projected/9829d8aa-db34-42b9-bf27-316fa4cc86f1-kube-api-access-vzc27\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.202677 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.202478 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.202677 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.202510 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.202677 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.202558 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.202677 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.202624 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.202677 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.202648 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.203017 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.202702 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.203017 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.202785 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.203017 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.202864 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.312848 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.309814 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.321790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.319973 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.321790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.320081 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzc27\" (UniqueName: \"kubernetes.io/projected/9829d8aa-db34-42b9-bf27-316fa4cc86f1-kube-api-access-vzc27\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.321790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.320116 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.321790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.320146 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.321790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.320198 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.321790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.320274 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.321790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.320302 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.321790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.320362 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.321790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.320434 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.321790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.320827 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.322486 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.321811 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.322486 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.322106 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.326788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.322695 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.368790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.365200 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.372787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.371882 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzc27\" (UniqueName: \"kubernetes.io/projected/9829d8aa-db34-42b9-bf27-316fa4cc86f1-kube-api-access-vzc27\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.414792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.409437 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.414792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.412213 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.650714 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.650547 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:54.670038 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.669987 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-sb5c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:49:54.670186 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.670085 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" podUID="a6fc6562-45cb-47db-98ab-415e184c71b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 20:49:54.999286 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:54.999259 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24"] Oct 30 20:49:55.001331 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:49:55.001298 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9829d8aa_db34_42b9_bf27_316fa4cc86f1.slice/crio-87bed5b15961bc7e983185ca6ac862ae65cffc9e00decfe56868dd53272ce3ca WatchSource:0}: Error finding container 87bed5b15961bc7e983185ca6ac862ae65cffc9e00decfe56868dd53272ce3ca: Status 404 returned error can't find the container with id 87bed5b15961bc7e983185ca6ac862ae65cffc9e00decfe56868dd53272ce3ca Oct 30 20:49:55.003754 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:55.003723 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:55.003892 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:55.003823 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:55.003892 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:55.003864 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:55.040964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:55.040938 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:55.041321 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:55.041304 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:55.042142 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:55.042119 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:49:55.601323 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:55.601281 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" event={"ID":"9829d8aa-db34-42b9-bf27-316fa4cc86f1","Type":"ContainerStarted","Data":"6e794d063e146fd1b52b97fb8015449ba5401e431d304b0d7929ed8fa181d2e8"} Oct 30 20:49:55.601492 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:55.601350 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" event={"ID":"9829d8aa-db34-42b9-bf27-316fa4cc86f1","Type":"ContainerStarted","Data":"87bed5b15961bc7e983185ca6ac862ae65cffc9e00decfe56868dd53272ce3ca"} Oct 30 20:49:55.642169 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:55.642102 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" podStartSLOduration=2.642083678 podStartE2EDuration="2.642083678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:55.64037526 +0000 UTC m=+700.820271770" watchObservedRunningTime="2026-10-30 20:49:55.642083678 +0000 UTC m=+700.821980186" Oct 30 20:49:55.650896 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:55.650864 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:49:55.668199 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:55.668165 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-sb5c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:49:55.668355 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:55.668228 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" podUID="a6fc6562-45cb-47db-98ab-415e184c71b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 20:49:56.031839 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.028525 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx"] Oct 30 20:49:56.100961 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.100756 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx"] Oct 30 20:49:56.100961 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.100907 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.194390 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.194341 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.194547 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.194450 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.194547 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.194513 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.194625 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.194543 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.194625 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.194598 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794057dc-d92a-4c15-8003-d3e8132297da-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.194696 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.194637 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t2d9\" (UniqueName: \"kubernetes.io/projected/794057dc-d92a-4c15-8003-d3e8132297da-kube-api-access-5t2d9\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.194696 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.194665 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794057dc-d92a-4c15-8003-d3e8132297da-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.194810 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.194793 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.194909 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.194897 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794057dc-d92a-4c15-8003-d3e8132297da-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.297577 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.297468 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794057dc-d92a-4c15-8003-d3e8132297da-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.297722 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.297588 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.297722 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.297682 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794057dc-d92a-4c15-8003-d3e8132297da-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.297806 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.297740 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.297806 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.297800 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.297874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.297830 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.297874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.297860 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.297934 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.297883 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794057dc-d92a-4c15-8003-d3e8132297da-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.297934 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.297910 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t2d9\" (UniqueName: \"kubernetes.io/projected/794057dc-d92a-4c15-8003-d3e8132297da-kube-api-access-5t2d9\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.298286 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.298262 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.298445 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.298428 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.298832 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.298562 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.298914 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.298886 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.299669 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.299632 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794057dc-d92a-4c15-8003-d3e8132297da-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.301235 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.301212 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794057dc-d92a-4c15-8003-d3e8132297da-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.301741 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.301721 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.360789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.360410 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794057dc-d92a-4c15-8003-d3e8132297da-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.362800 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.362518 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t2d9\" (UniqueName: \"kubernetes.io/projected/794057dc-d92a-4c15-8003-d3e8132297da-kube-api-access-5t2d9\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.484223 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.483010 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:56.667952 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.667917 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:49:56.668221 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.668191 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" podUID="9829d8aa-db34-42b9-bf27-316fa4cc86f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 20:49:56.676277 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.668878 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-sb5c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:49:56.676277 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:56.676159 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" podUID="a6fc6562-45cb-47db-98ab-415e184c71b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 20:49:57.365991 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:57.354599 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx"] Oct 30 20:49:57.438805 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:57.431589 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:57.438805 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:57.431693 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:57.438805 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:57.431742 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:49:57.658355 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:57.656105 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:49:57.658355 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:57.656166 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" podUID="9829d8aa-db34-42b9-bf27-316fa4cc86f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 20:49:57.671895 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:57.671727 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-sb5c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:49:57.671895 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:57.671797 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" podUID="a6fc6562-45cb-47db-98ab-415e184c71b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 20:49:57.689944 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:57.689871 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" event={"ID":"794057dc-d92a-4c15-8003-d3e8132297da","Type":"ContainerStarted","Data":"ecebe43270f24fa3e2a333f2e30b4ffc916a471b703424a9c687bc9e834eca93"} Oct 30 20:49:58.262926 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:58.262871 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:49:58.264293 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:58.264260 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 20:49:58.654350 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:58.654252 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:49:58.654957 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:58.654924 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" podUID="9829d8aa-db34-42b9-bf27-316fa4cc86f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 20:49:58.668719 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:58.668687 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-sb5c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:49:58.668978 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:58.668948 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" podUID="a6fc6562-45cb-47db-98ab-415e184c71b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 20:49:58.741792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:58.739951 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" event={"ID":"794057dc-d92a-4c15-8003-d3e8132297da","Type":"ContainerStarted","Data":"1f64d311a40d18931f63c5037f1375d1ea0e552d26dc166786963a74d0e63c74"} Oct 30 20:49:59.489792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:59.489056 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:49:59.492806 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:59.490818 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:49:59.492806 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:59.490874 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" podUID="794057dc-d92a-4c15-8003-d3e8132297da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 20:49:59.656791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:59.656163 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:49:59.656791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:59.656236 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" podUID="9829d8aa-db34-42b9-bf27-316fa4cc86f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 20:49:59.669790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:59.668130 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-sb5c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:49:59.669790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:49:59.668186 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" podUID="a6fc6562-45cb-47db-98ab-415e184c71b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 20:50:00.128895 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:00.128846 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:50:00.484244 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:00.484201 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:50:00.484566 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:00.484535 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" podUID="794057dc-d92a-4c15-8003-d3e8132297da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 20:50:00.651789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:00.651202 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:50:00.651789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:00.651280 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" podUID="9829d8aa-db34-42b9-bf27-316fa4cc86f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 20:50:00.668587 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:00.668451 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-sb5c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:50:00.668587 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:00.668527 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" podUID="a6fc6562-45cb-47db-98ab-415e184c71b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 20:50:00.887844 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:00.878492 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" podStartSLOduration=5.878470201 podStartE2EDuration="5.878470201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:49:58.821261955 +0000 UTC m=+704.001158466" watchObservedRunningTime="2026-10-30 20:50:00.878470201 +0000 UTC m=+706.058366711" Oct 30 20:50:00.924798 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:00.916261 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf"] Oct 30 20:50:01.278734 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:01.278688 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Oct 30 20:50:01.487804 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:01.484567 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:50:01.487804 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:01.484625 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" podUID="794057dc-d92a-4c15-8003-d3e8132297da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 20:50:01.654328 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:01.654225 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:50:01.654606 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:01.654565 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" podUID="9829d8aa-db34-42b9-bf27-316fa4cc86f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 20:50:01.669861 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:01.668137 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-sb5c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:50:01.669861 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:01.668209 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" podUID="a6fc6562-45cb-47db-98ab-415e184c71b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 20:50:02.484360 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:02.484330 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:50:02.484614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:02.484593 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" podUID="794057dc-d92a-4c15-8003-d3e8132297da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 20:50:02.654789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:02.654176 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 20:50:02.654789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:02.654240 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" podUID="9829d8aa-db34-42b9-bf27-316fa4cc86f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 20:50:02.669414 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:02.669379 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-sb5c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 20:50:02.669672 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:02.669644 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" podUID="a6fc6562-45cb-47db-98ab-415e184c71b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 20:50:03.484589 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:03.484550 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:50:03.485245 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:03.485213 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" podUID="794057dc-d92a-4c15-8003-d3e8132297da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 20:50:04.161794 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.160868 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m"] Oct 30 20:50:04.172161 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.172125 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" Oct 30 20:50:04.178143 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.178114 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m"] Oct 30 20:50:04.341792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.340729 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqfzc\" (UniqueName: \"kubernetes.io/projected/198c97d6-5b51-47d5-b034-c220fafb7a2b-kube-api-access-cqfzc\") pod \"payload-processing-external-model-5d5f6f949f-mmc9m\" (UID: \"198c97d6-5b51-47d5-b034-c220fafb7a2b\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" Oct 30 20:50:04.341792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.340823 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198c97d6-5b51-47d5-b034-c220fafb7a2b-plugins-config-volume\") pod \"payload-processing-external-model-5d5f6f949f-mmc9m\" (UID: \"198c97d6-5b51-47d5-b034-c220fafb7a2b\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" Oct 30 20:50:04.341792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.340891 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/198c97d6-5b51-47d5-b034-c220fafb7a2b-routing-overlay-volume\") pod \"payload-processing-external-model-5d5f6f949f-mmc9m\" (UID: \"198c97d6-5b51-47d5-b034-c220fafb7a2b\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" Oct 30 20:50:04.341792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.340945 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/198c97d6-5b51-47d5-b034-c220fafb7a2b-provider-credentials\") pod \"payload-processing-external-model-5d5f6f949f-mmc9m\" (UID: \"198c97d6-5b51-47d5-b034-c220fafb7a2b\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" Oct 30 20:50:04.441758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.441714 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqfzc\" (UniqueName: \"kubernetes.io/projected/198c97d6-5b51-47d5-b034-c220fafb7a2b-kube-api-access-cqfzc\") pod \"payload-processing-external-model-5d5f6f949f-mmc9m\" (UID: \"198c97d6-5b51-47d5-b034-c220fafb7a2b\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" Oct 30 20:50:04.442390 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.442368 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198c97d6-5b51-47d5-b034-c220fafb7a2b-plugins-config-volume\") pod \"payload-processing-external-model-5d5f6f949f-mmc9m\" (UID: \"198c97d6-5b51-47d5-b034-c220fafb7a2b\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" Oct 30 20:50:04.442589 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.442574 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/198c97d6-5b51-47d5-b034-c220fafb7a2b-routing-overlay-volume\") pod \"payload-processing-external-model-5d5f6f949f-mmc9m\" (UID: \"198c97d6-5b51-47d5-b034-c220fafb7a2b\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" Oct 30 20:50:04.442730 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.442717 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/198c97d6-5b51-47d5-b034-c220fafb7a2b-provider-credentials\") pod \"payload-processing-external-model-5d5f6f949f-mmc9m\" (UID: \"198c97d6-5b51-47d5-b034-c220fafb7a2b\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" Oct 30 20:50:04.444953 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.444692 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198c97d6-5b51-47d5-b034-c220fafb7a2b-plugins-config-volume\") pod \"payload-processing-external-model-5d5f6f949f-mmc9m\" (UID: \"198c97d6-5b51-47d5-b034-c220fafb7a2b\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" Oct 30 20:50:04.445403 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.445346 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/198c97d6-5b51-47d5-b034-c220fafb7a2b-routing-overlay-volume\") pod \"payload-processing-external-model-5d5f6f949f-mmc9m\" (UID: \"198c97d6-5b51-47d5-b034-c220fafb7a2b\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" Oct 30 20:50:04.448801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.448574 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/198c97d6-5b51-47d5-b034-c220fafb7a2b-provider-credentials\") pod \"payload-processing-external-model-5d5f6f949f-mmc9m\" (UID: \"198c97d6-5b51-47d5-b034-c220fafb7a2b\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" Oct 30 20:50:04.453020 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.452997 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqfzc\" (UniqueName: \"kubernetes.io/projected/198c97d6-5b51-47d5-b034-c220fafb7a2b-kube-api-access-cqfzc\") pod \"payload-processing-external-model-5d5f6f949f-mmc9m\" (UID: \"198c97d6-5b51-47d5-b034-c220fafb7a2b\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" Oct 30 20:50:04.484384 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.483937 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:50:04.484384 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.484000 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" podUID="794057dc-d92a-4c15-8003-d3e8132297da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 20:50:04.494405 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.494375 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" Oct 30 20:50:04.647946 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.641274 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:50:04.647946 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.646134 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:50:04.677526 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.677457 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" Oct 30 20:50:04.698247 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.698138 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:50:04.698247 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.698198 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" podUID="9829d8aa-db34-42b9-bf27-316fa4cc86f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:50:04.982950 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:04.982924 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m"] Oct 30 20:50:05.272379 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:05.272305 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:50:05.272699 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:05.272681 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:50:05.273590 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:05.273561 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 20:50:05.488815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:05.483885 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 20:50:05.488815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:05.483941 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" podUID="794057dc-d92a-4c15-8003-d3e8132297da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 20:50:05.878042 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:05.878006 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" event={"ID":"198c97d6-5b51-47d5-b034-c220fafb7a2b","Type":"ContainerStarted","Data":"ac7903fa278860e661d3abc91372f233c7af49def1c58004cc80796f7fd42824"} Oct 30 20:50:05.878488 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:05.878072 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" event={"ID":"198c97d6-5b51-47d5-b034-c220fafb7a2b","Type":"ContainerStarted","Data":"7abd3ad23a7d244d230cd21df61263adf80877685bdf0bd30a6bf74532421bf9"} Oct 30 20:50:05.878488 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:05.878308 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" Oct 30 20:50:05.900170 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:05.900101 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" podStartSLOduration=1.900080422 podStartE2EDuration="1.900080422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:05.897390995 +0000 UTC m=+711.077287499" watchObservedRunningTime="2026-10-30 20:50:05.900080422 +0000 UTC m=+711.079976931" Oct 30 20:50:05.916714 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:05.916671 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" podUID="a6d9b62c-2a59-4223-805b-33263b17d846" containerName="payload-processing" containerID="cri-o://48197571f659be6b5eec296aa4d171170bf82d136f9cad5950e7362ab0efe53f" gracePeriod=25 Oct 30 20:50:06.219132 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.219092 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" Oct 30 20:50:06.284610 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.284567 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d9b62c-2a59-4223-805b-33263b17d846-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6d9b62c-2a59-4223-805b-33263b17d846-provider-credentials\") pod \"a6d9b62c-2a59-4223-805b-33263b17d846\" (UID: \"a6d9b62c-2a59-4223-805b-33263b17d846\") " Oct 30 20:50:06.284815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.284671 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6d9b62c-2a59-4223-805b-33263b17d846-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6d9b62c-2a59-4223-805b-33263b17d846-routing-overlay-volume\") pod \"a6d9b62c-2a59-4223-805b-33263b17d846\" (UID: \"a6d9b62c-2a59-4223-805b-33263b17d846\") " Oct 30 20:50:06.284815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.284730 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d9b62c-2a59-4223-805b-33263b17d846-kube-api-access-r77tz\" (UniqueName: \"kubernetes.io/projected/a6d9b62c-2a59-4223-805b-33263b17d846-kube-api-access-r77tz\") pod \"a6d9b62c-2a59-4223-805b-33263b17d846\" (UID: \"a6d9b62c-2a59-4223-805b-33263b17d846\") " Oct 30 20:50:06.284941 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.284888 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6d9b62c-2a59-4223-805b-33263b17d846-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d9b62c-2a59-4223-805b-33263b17d846-plugins-config-volume\") pod \"a6d9b62c-2a59-4223-805b-33263b17d846\" (UID: \"a6d9b62c-2a59-4223-805b-33263b17d846\") " Oct 30 20:50:06.285500 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.285170 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d9b62c-2a59-4223-805b-33263b17d846-routing-overlay-volume" pod "a6d9b62c-2a59-4223-805b-33263b17d846" (UID: "a6d9b62c-2a59-4223-805b-33263b17d846"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:50:06.285500 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.285468 4848 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a6d9b62c-2a59-4223-805b-33263b17d846-routing-overlay-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:50:06.285682 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.285563 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6d9b62c-2a59-4223-805b-33263b17d846-plugins-config-volume" pod "a6d9b62c-2a59-4223-805b-33263b17d846" (UID: "a6d9b62c-2a59-4223-805b-33263b17d846"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:50:06.291994 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.291927 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d9b62c-2a59-4223-805b-33263b17d846-provider-credentials" pod "a6d9b62c-2a59-4223-805b-33263b17d846" (UID: "a6d9b62c-2a59-4223-805b-33263b17d846"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:50:06.292284 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.292246 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d9b62c-2a59-4223-805b-33263b17d846-kube-api-access-r77tz" pod "a6d9b62c-2a59-4223-805b-33263b17d846" (UID: "a6d9b62c-2a59-4223-805b-33263b17d846"). InnerVolumeSpecName "kube-api-access-r77tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:50:06.387069 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.387032 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r77tz\" (UniqueName: \"kubernetes.io/projected/a6d9b62c-2a59-4223-805b-33263b17d846-kube-api-access-r77tz\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:50:06.387069 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.387061 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6d9b62c-2a59-4223-805b-33263b17d846-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:50:06.387069 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.387073 4848 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a6d9b62c-2a59-4223-805b-33263b17d846-provider-credentials\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:50:06.765055 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.764321 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:50:06.765256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.765107 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:50:06.766649 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.766418 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:50:06.885331 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.885295 4848 generic.go:356] "Generic (PLEG): container finished" podID="a6d9b62c-2a59-4223-805b-33263b17d846" containerID="48197571f659be6b5eec296aa4d171170bf82d136f9cad5950e7362ab0efe53f" exitCode=0 Oct 30 20:50:06.885807 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.885342 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" Oct 30 20:50:06.885807 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.885372 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" event={"ID":"a6d9b62c-2a59-4223-805b-33263b17d846","Type":"ContainerDied","Data":"48197571f659be6b5eec296aa4d171170bf82d136f9cad5950e7362ab0efe53f"} Oct 30 20:50:06.885807 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.885432 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf" event={"ID":"a6d9b62c-2a59-4223-805b-33263b17d846","Type":"ContainerDied","Data":"20a9d0e31fe38f611e84d7fc47d7e39d8c25a1875e10fa8c8d657e5398da4bd7"} Oct 30 20:50:06.885807 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.885456 4848 scope.go:122] "RemoveContainer" containerID="48197571f659be6b5eec296aa4d171170bf82d136f9cad5950e7362ab0efe53f" Oct 30 20:50:06.896741 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.896716 4848 scope.go:122] "RemoveContainer" containerID="48197571f659be6b5eec296aa4d171170bf82d136f9cad5950e7362ab0efe53f" Oct 30 20:50:06.897138 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:50:06.897115 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48197571f659be6b5eec296aa4d171170bf82d136f9cad5950e7362ab0efe53f\": container with ID starting with 48197571f659be6b5eec296aa4d171170bf82d136f9cad5950e7362ab0efe53f not found: ID does not exist" containerID="48197571f659be6b5eec296aa4d171170bf82d136f9cad5950e7362ab0efe53f" Oct 30 20:50:06.897204 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.897147 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48197571f659be6b5eec296aa4d171170bf82d136f9cad5950e7362ab0efe53f"} err="failed to get container status \"48197571f659be6b5eec296aa4d171170bf82d136f9cad5950e7362ab0efe53f\": rpc error: code = NotFound desc = could not find container \"48197571f659be6b5eec296aa4d171170bf82d136f9cad5950e7362ab0efe53f\": container with ID starting with 48197571f659be6b5eec296aa4d171170bf82d136f9cad5950e7362ab0efe53f not found: ID does not exist" Oct 30 20:50:06.913622 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.913593 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf"] Oct 30 20:50:06.918878 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.918853 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-d65b6b7bb-k98hf"] Oct 30 20:50:06.936733 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:06.935883 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d9b62c-2a59-4223-805b-33263b17d846" path="/var/lib/kubelet/pods/a6d9b62c-2a59-4223-805b-33263b17d846/volumes" Oct 30 20:50:16.890018 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:16.889989 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" Oct 30 20:50:17.894890 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:17.894854 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4"] Oct 30 20:50:17.895402 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:17.895388 4848 cpu_manager.go:414] "containerMap: removing container" podUID="a6d9b62c-2a59-4223-805b-33263b17d846" containerName="payload-processing" Oct 30 20:50:17.895448 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:17.895404 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d9b62c-2a59-4223-805b-33263b17d846" containerName="payload-processing" Oct 30 20:50:17.895495 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:17.895478 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d9b62c-2a59-4223-805b-33263b17d846" containerName="payload-processing" Oct 30 20:50:17.898626 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:17.898604 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:17.907239 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:17.907218 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-f9a819\"/\"distinct-2-w3-f9a819-kserve-self-signed-certs\"" Oct 30 20:50:17.909455 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:17.909433 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4"] Oct 30 20:50:17.990572 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:17.990540 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-home\") pod \"distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:17.990572 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:17.990578 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-model-cache\") pod \"distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:17.990829 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:17.990608 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-dshm\") pod \"distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:17.990829 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:17.990719 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/130328d3-b4f3-4123-9116-c5e462b0c8dc-tls-certs\") pod \"distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:17.990829 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:17.990752 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5h96\" (UniqueName: \"kubernetes.io/projected/130328d3-b4f3-4123-9116-c5e462b0c8dc-kube-api-access-t5h96\") pod \"distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:17.990955 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:17.990918 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-tmp-dir\") pod \"distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:18.093283 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:18.093233 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-home\") pod \"distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:18.093496 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:18.093296 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-model-cache\") pod \"distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:18.093496 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:18.093340 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-dshm\") pod \"distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:18.093496 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:18.093390 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/130328d3-b4f3-4123-9116-c5e462b0c8dc-tls-certs\") pod \"distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:18.093496 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:18.093421 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5h96\" (UniqueName: \"kubernetes.io/projected/130328d3-b4f3-4123-9116-c5e462b0c8dc-kube-api-access-t5h96\") pod \"distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:18.093718 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:18.093569 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-tmp-dir\") pod \"distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:18.094073 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:18.094048 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-tmp-dir\") pod \"distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:18.094318 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:18.094298 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-home\") pod \"distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:18.094584 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:18.094556 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-model-cache\") pod \"distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:18.101300 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:18.101254 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/130328d3-b4f3-4123-9116-c5e462b0c8dc-tls-certs\") pod \"distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:18.109740 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:18.109703 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-dshm\") pod \"distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:18.112709 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:18.112676 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5h96\" (UniqueName: \"kubernetes.io/projected/130328d3-b4f3-4123-9116-c5e462b0c8dc-kube-api-access-t5h96\") pod \"distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:18.211897 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:18.211861 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:18.492473 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:50:18.492432 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod130328d3_b4f3_4123_9116_c5e462b0c8dc.slice/crio-9cdd6b55193a665f551ac7d7eb5fad35f09acc4cc00eb9e2ba121bf87a2d5fa8 WatchSource:0}: Error finding container 9cdd6b55193a665f551ac7d7eb5fad35f09acc4cc00eb9e2ba121bf87a2d5fa8: Status 404 returned error can't find the container with id 9cdd6b55193a665f551ac7d7eb5fad35f09acc4cc00eb9e2ba121bf87a2d5fa8 Oct 30 20:50:18.492680 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:18.492654 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4"] Oct 30 20:50:19.087435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:19.087393 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" event={"ID":"130328d3-b4f3-4123-9116-c5e462b0c8dc","Type":"ContainerStarted","Data":"4e3e8b05c29097257261f9cfde29e264074efc7fe8741b83da96be17007db80f"} Oct 30 20:50:19.087435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:19.087436 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" event={"ID":"130328d3-b4f3-4123-9116-c5e462b0c8dc","Type":"ContainerStarted","Data":"9cdd6b55193a665f551ac7d7eb5fad35f09acc4cc00eb9e2ba121bf87a2d5fa8"} Oct 30 20:50:19.113303 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:19.113255 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podStartSLOduration=2.113240823 podStartE2EDuration="2.113240823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:19.111946771 +0000 UTC m=+724.291843278" watchObservedRunningTime="2026-10-30 20:50:19.113240823 +0000 UTC m=+724.293137329" Oct 30 20:50:19.212491 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:19.212465 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:22.512829 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:50:22.512761 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:50:28.212561 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:28.212521 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:28.233018 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:28.232980 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:29.267790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:29.267131 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:50:30.941459 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:30.941409 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:50:31.132429 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:31.132315 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:50:31.162692 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:31.162636 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:50:31.266154 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:31.264133 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:50:31.944823 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:31.944760 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 20:50:31.950701 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:31.950673 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:50:32.191789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:32.182941 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:50:32.398116 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:32.398027 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-mfvmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:50:32.398116 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:32.398104 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" podUID="0226107b-80ef-4c6e-888c-f89f73108c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:50:34.519682 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:34.519644 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m"] Oct 30 20:50:39.521611 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:39.521565 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" podUID="198c97d6-5b51-47d5-b034-c220fafb7a2b" containerName="payload-processing" containerID="cri-o://ac7903fa278860e661d3abc91372f233c7af49def1c58004cc80796f7fd42824" gracePeriod=25 Oct 30 20:50:39.781621 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:39.781561 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" Oct 30 20:50:39.843499 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:39.843467 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/198c97d6-5b51-47d5-b034-c220fafb7a2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198c97d6-5b51-47d5-b034-c220fafb7a2b-plugins-config-volume\") pod \"198c97d6-5b51-47d5-b034-c220fafb7a2b\" (UID: \"198c97d6-5b51-47d5-b034-c220fafb7a2b\") " Oct 30 20:50:39.843692 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:39.843509 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/198c97d6-5b51-47d5-b034-c220fafb7a2b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/198c97d6-5b51-47d5-b034-c220fafb7a2b-routing-overlay-volume\") pod \"198c97d6-5b51-47d5-b034-c220fafb7a2b\" (UID: \"198c97d6-5b51-47d5-b034-c220fafb7a2b\") " Oct 30 20:50:39.843793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:39.843757 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198c97d6-5b51-47d5-b034-c220fafb7a2b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/198c97d6-5b51-47d5-b034-c220fafb7a2b-provider-credentials\") pod \"198c97d6-5b51-47d5-b034-c220fafb7a2b\" (UID: \"198c97d6-5b51-47d5-b034-c220fafb7a2b\") " Oct 30 20:50:39.843862 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:39.843826 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198c97d6-5b51-47d5-b034-c220fafb7a2b-kube-api-access-cqfzc\" (UniqueName: \"kubernetes.io/projected/198c97d6-5b51-47d5-b034-c220fafb7a2b-kube-api-access-cqfzc\") pod \"198c97d6-5b51-47d5-b034-c220fafb7a2b\" (UID: \"198c97d6-5b51-47d5-b034-c220fafb7a2b\") " Oct 30 20:50:39.843949 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:39.843919 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/198c97d6-5b51-47d5-b034-c220fafb7a2b-plugins-config-volume" pod "198c97d6-5b51-47d5-b034-c220fafb7a2b" (UID: "198c97d6-5b51-47d5-b034-c220fafb7a2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:50:39.844007 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:39.843956 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/198c97d6-5b51-47d5-b034-c220fafb7a2b-routing-overlay-volume" pod "198c97d6-5b51-47d5-b034-c220fafb7a2b" (UID: "198c97d6-5b51-47d5-b034-c220fafb7a2b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:50:39.844300 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:39.844281 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/198c97d6-5b51-47d5-b034-c220fafb7a2b-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:50:39.844405 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:39.844305 4848 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/198c97d6-5b51-47d5-b034-c220fafb7a2b-routing-overlay-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:50:39.846266 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:39.846236 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198c97d6-5b51-47d5-b034-c220fafb7a2b-kube-api-access-cqfzc" pod "198c97d6-5b51-47d5-b034-c220fafb7a2b" (UID: "198c97d6-5b51-47d5-b034-c220fafb7a2b"). InnerVolumeSpecName "kube-api-access-cqfzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:50:39.846368 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:39.846318 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198c97d6-5b51-47d5-b034-c220fafb7a2b-provider-credentials" pod "198c97d6-5b51-47d5-b034-c220fafb7a2b" (UID: "198c97d6-5b51-47d5-b034-c220fafb7a2b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:50:39.950675 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:39.950638 4848 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/198c97d6-5b51-47d5-b034-c220fafb7a2b-provider-credentials\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:50:39.957367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:39.957318 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqfzc\" (UniqueName: \"kubernetes.io/projected/198c97d6-5b51-47d5-b034-c220fafb7a2b-kube-api-access-cqfzc\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:50:40.491591 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:40.491553 4848 generic.go:356] "Generic (PLEG): container finished" podID="198c97d6-5b51-47d5-b034-c220fafb7a2b" containerID="ac7903fa278860e661d3abc91372f233c7af49def1c58004cc80796f7fd42824" exitCode=0 Oct 30 20:50:40.491789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:40.491734 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" event={"ID":"198c97d6-5b51-47d5-b034-c220fafb7a2b","Type":"ContainerDied","Data":"ac7903fa278860e661d3abc91372f233c7af49def1c58004cc80796f7fd42824"} Oct 30 20:50:40.491861 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:40.491819 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" event={"ID":"198c97d6-5b51-47d5-b034-c220fafb7a2b","Type":"ContainerDied","Data":"7abd3ad23a7d244d230cd21df61263adf80877685bdf0bd30a6bf74532421bf9"} Oct 30 20:50:40.491861 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:40.491854 4848 scope.go:122] "RemoveContainer" containerID="ac7903fa278860e661d3abc91372f233c7af49def1c58004cc80796f7fd42824" Oct 30 20:50:40.492049 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:40.492034 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m" Oct 30 20:50:40.514141 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:40.514116 4848 scope.go:122] "RemoveContainer" containerID="ac7903fa278860e661d3abc91372f233c7af49def1c58004cc80796f7fd42824" Oct 30 20:50:40.515445 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:50:40.515413 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac7903fa278860e661d3abc91372f233c7af49def1c58004cc80796f7fd42824\": container with ID starting with ac7903fa278860e661d3abc91372f233c7af49def1c58004cc80796f7fd42824 not found: ID does not exist" containerID="ac7903fa278860e661d3abc91372f233c7af49def1c58004cc80796f7fd42824" Oct 30 20:50:40.515581 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:40.515454 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac7903fa278860e661d3abc91372f233c7af49def1c58004cc80796f7fd42824"} err="failed to get container status \"ac7903fa278860e661d3abc91372f233c7af49def1c58004cc80796f7fd42824\": rpc error: code = NotFound desc = could not find container \"ac7903fa278860e661d3abc91372f233c7af49def1c58004cc80796f7fd42824\": container with ID starting with ac7903fa278860e661d3abc91372f233c7af49def1c58004cc80796f7fd42824 not found: ID does not exist" Oct 30 20:50:40.527861 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:40.527832 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m"] Oct 30 20:50:40.537513 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:40.537487 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f6f949f-mmc9m"] Oct 30 20:50:40.940567 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:40.940532 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="198c97d6-5b51-47d5-b034-c220fafb7a2b" path="/var/lib/kubelet/pods/198c97d6-5b51-47d5-b034-c220fafb7a2b/volumes" Oct 30 20:50:47.123971 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:47.123867 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:50:47.230920 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:47.230874 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:50:47.541788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:47.535867 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:50:47.541788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:47.535932 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" podUID="16e788c7-69cc-48cb-959a-62c4763c2698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:50:47.947980 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:47.947924 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:50:48.131792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:48.128848 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:50:48.131792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:48.129279 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:50:48.695793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:48.671363 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:50:50.934579 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:50.934542 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr"] Oct 30 20:50:50.935344 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:50.935324 4848 cpu_manager.go:414] "containerMap: removing container" podUID="198c97d6-5b51-47d5-b034-c220fafb7a2b" containerName="payload-processing" Oct 30 20:50:50.935397 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:50.935348 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198c97d6-5b51-47d5-b034-c220fafb7a2b" containerName="payload-processing" Oct 30 20:50:50.935526 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:50.935513 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="198c97d6-5b51-47d5-b034-c220fafb7a2b" containerName="payload-processing" Oct 30 20:50:50.939253 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:50.939229 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:50.950150 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:50.950130 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-bc102e\"/\"distinct-w0-bc102e-kserve-self-signed-certs\"" Oct 30 20:50:50.952026 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:50.952006 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr"] Oct 30 20:50:51.017558 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.017522 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg5hl\" (UniqueName: \"kubernetes.io/projected/9184c819-a720-46b2-992b-65a4087be757-kube-api-access-mg5hl\") pod \"distinct-w0-bc102e-kserve-7d4898f6d6-65rhr\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:51.017739 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.017574 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-dshm\") pod \"distinct-w0-bc102e-kserve-7d4898f6d6-65rhr\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:51.017739 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.017697 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-model-cache\") pod \"distinct-w0-bc102e-kserve-7d4898f6d6-65rhr\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:51.017739 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.017731 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-tmp-dir\") pod \"distinct-w0-bc102e-kserve-7d4898f6d6-65rhr\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:51.017911 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.017823 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9184c819-a720-46b2-992b-65a4087be757-tls-certs\") pod \"distinct-w0-bc102e-kserve-7d4898f6d6-65rhr\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:51.017911 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.017871 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-home\") pod \"distinct-w0-bc102e-kserve-7d4898f6d6-65rhr\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:51.121917 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.121876 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9184c819-a720-46b2-992b-65a4087be757-tls-certs\") pod \"distinct-w0-bc102e-kserve-7d4898f6d6-65rhr\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:51.122179 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.122156 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-home\") pod \"distinct-w0-bc102e-kserve-7d4898f6d6-65rhr\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:51.122485 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.122463 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg5hl\" (UniqueName: \"kubernetes.io/projected/9184c819-a720-46b2-992b-65a4087be757-kube-api-access-mg5hl\") pod \"distinct-w0-bc102e-kserve-7d4898f6d6-65rhr\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:51.122719 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.122700 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-dshm\") pod \"distinct-w0-bc102e-kserve-7d4898f6d6-65rhr\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:51.122897 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.122877 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-model-cache\") pod \"distinct-w0-bc102e-kserve-7d4898f6d6-65rhr\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:51.123035 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.123018 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-tmp-dir\") pod \"distinct-w0-bc102e-kserve-7d4898f6d6-65rhr\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:51.123241 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.123216 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-home\") pod \"distinct-w0-bc102e-kserve-7d4898f6d6-65rhr\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:51.123801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.123498 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-tmp-dir\") pod \"distinct-w0-bc102e-kserve-7d4898f6d6-65rhr\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:51.124187 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.124168 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-model-cache\") pod \"distinct-w0-bc102e-kserve-7d4898f6d6-65rhr\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:51.130485 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.130457 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9184c819-a720-46b2-992b-65a4087be757-tls-certs\") pod \"distinct-w0-bc102e-kserve-7d4898f6d6-65rhr\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:51.132942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.132915 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-dshm\") pod \"distinct-w0-bc102e-kserve-7d4898f6d6-65rhr\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:51.140129 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.140090 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg5hl\" (UniqueName: \"kubernetes.io/projected/9184c819-a720-46b2-992b-65a4087be757-kube-api-access-mg5hl\") pod \"distinct-w0-bc102e-kserve-7d4898f6d6-65rhr\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:51.256684 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.256593 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:51.543404 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.543376 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr"] Oct 30 20:50:51.545625 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:50:51.545596 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9184c819_a720_46b2_992b_65a4087be757.slice/crio-38d5c6b7d2c967e034e0112875f70289825a7e37f92970ca46351b508fb18ad9 WatchSource:0}: Error finding container 38d5c6b7d2c967e034e0112875f70289825a7e37f92970ca46351b508fb18ad9: Status 404 returned error can't find the container with id 38d5c6b7d2c967e034e0112875f70289825a7e37f92970ca46351b508fb18ad9 Oct 30 20:50:51.795107 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.795002 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" event={"ID":"9184c819-a720-46b2-992b-65a4087be757","Type":"ContainerStarted","Data":"bd747dd58ce1109aea46f23c1fb0d6057bbd979a8a4743420c626b871af41b73"} Oct 30 20:50:51.795107 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.795058 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" event={"ID":"9184c819-a720-46b2-992b-65a4087be757","Type":"ContainerStarted","Data":"38d5c6b7d2c967e034e0112875f70289825a7e37f92970ca46351b508fb18ad9"} Oct 30 20:50:51.816289 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:51.816216 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podStartSLOduration=1.8161913840000001 podStartE2EDuration="1.816191384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:51.811815444 +0000 UTC m=+756.991711963" watchObservedRunningTime="2026-10-30 20:50:51.816191384 +0000 UTC m=+756.996087896" Oct 30 20:50:52.256861 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:52.256830 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:50:52.575134 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:50:52.575036 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:50:56.092159 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.092125 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb"] Oct 30 20:50:56.096322 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.096299 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.113493 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.113468 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb"] Oct 30 20:50:56.194590 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.194558 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.194758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.194597 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.194758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.194638 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9ndm\" (UniqueName: \"kubernetes.io/projected/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-kube-api-access-b9ndm\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.194758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.194725 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.194972 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.194878 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.194972 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.194966 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.195074 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.194992 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.195134 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.195103 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.195205 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.195185 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.295888 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.295845 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.296027 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.295903 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.296027 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.295935 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.296027 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.295957 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9ndm\" (UniqueName: \"kubernetes.io/projected/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-kube-api-access-b9ndm\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.296027 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.296010 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.296194 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.296103 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.296194 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.296154 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.296194 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.296179 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.296352 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.296258 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.296434 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.296409 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.296730 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.296707 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.296730 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.296733 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.296925 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.296748 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.296925 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.296823 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.298357 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.298332 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.298597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.298578 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.305017 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.304995 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9ndm\" (UniqueName: \"kubernetes.io/projected/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-kube-api-access-b9ndm\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.310688 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.310665 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.411304 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.411206 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:56.716656 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.716626 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb"] Oct 30 20:50:56.717262 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:50:56.717231 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e6a0bd_600e_46d0_b8c5_226aa8fbf72d.slice/crio-2974bf64e1fd92e77ece5c2099ec121696c836100aed7ddba067bd821ae25506 WatchSource:0}: Error finding container 2974bf64e1fd92e77ece5c2099ec121696c836100aed7ddba067bd821ae25506: Status 404 returned error can't find the container with id 2974bf64e1fd92e77ece5c2099ec121696c836100aed7ddba067bd821ae25506 Oct 30 20:50:56.719690 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.719655 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:56.719804 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.719731 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:56.719804 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.719780 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:50:56.827511 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.827474 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" event={"ID":"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d","Type":"ContainerStarted","Data":"a20a378097eceae6fc8987bbc2b885b752fb17ef981f1359169809adfbcf8bbc"} Oct 30 20:50:56.827699 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.827571 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" event={"ID":"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d","Type":"ContainerStarted","Data":"2974bf64e1fd92e77ece5c2099ec121696c836100aed7ddba067bd821ae25506"} Oct 30 20:50:56.857377 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:56.857316 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" podStartSLOduration=0.857301604 podStartE2EDuration="857.301604ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:50:56.854207603 +0000 UTC m=+762.034104126" watchObservedRunningTime="2026-10-30 20:50:56.857301604 +0000 UTC m=+762.037198111" Oct 30 20:50:57.411746 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:57.411714 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:50:57.413175 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:57.413151 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:50:57.413297 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:57.413196 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" podUID="f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 20:50:58.417263 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:58.417220 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:50:58.417956 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:58.417925 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" podUID="f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 20:50:59.412167 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:59.412128 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:50:59.412429 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:50:59.412401 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" podUID="f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 20:51:00.420283 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:00.420242 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:51:00.421054 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:00.421017 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" podUID="f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 20:51:01.260242 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:01.260206 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:51:01.420209 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:01.420171 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:51:01.420522 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:01.420495 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" podUID="f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 20:51:01.603306 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:01.603220 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:51:01.979668 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:01.979637 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:51:02.001250 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.001217 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz"] Oct 30 20:51:02.048995 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.048961 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz"] Oct 30 20:51:02.084897 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.059465 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmxd8\" (UniqueName: \"kubernetes.io/projected/3a9a5360-6f89-4b28-9638-a56fdb01389f-kube-api-access-fmxd8\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.085201 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.085173 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.086208 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.060000 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.105928 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.105829 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.106153 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.106133 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.106334 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.106318 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.106513 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.106497 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.206042 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.205912 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a9a5360-6f89-4b28-9638-a56fdb01389f-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.206561 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.206528 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.317795 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.315467 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.317795 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.300117 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j"] Oct 30 20:51:02.364829 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.362492 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j"] Oct 30 20:51:02.364829 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.362632 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.416836 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.416804 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.417122 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.417099 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.417269 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.417251 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.417400 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.417384 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.417572 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.417557 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmxd8\" (UniqueName: \"kubernetes.io/projected/3a9a5360-6f89-4b28-9638-a56fdb01389f-kube-api-access-fmxd8\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.417696 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.417681 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.417864 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.417843 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.418002 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.417987 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.418134 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.418102 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.418250 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.418235 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.418344 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.418332 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.428912 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.428869 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.429417 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.429395 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a9a5360-6f89-4b28-9638-a56fdb01389f-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.429580 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.429553 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.429705 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.429690 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.429901 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.429885 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.430093 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.430062 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.444983 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.444858 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jggb7\" (UniqueName: \"kubernetes.io/projected/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-kube-api-access-jggb7\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.475590 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.446235 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.475952 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.446725 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.485543 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.454514 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a9a5360-6f89-4b28-9638-a56fdb01389f-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.485802 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.454793 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.494879 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.455051 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.495105 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.477895 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:51:02.495278 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.495248 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" podUID="f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 20:51:02.586152 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.576917 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.612000 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.586509 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jggb7\" (UniqueName: \"kubernetes.io/projected/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-kube-api-access-jggb7\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.635787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.625401 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.635787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.625461 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.635787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.625505 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.635787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.625676 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.635787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.625720 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.635787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.625795 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.635787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.625830 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.787800 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.597742 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.789795 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.788263 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.789795 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.788598 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.789795 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.789614 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jggb7\" (UniqueName: \"kubernetes.io/projected/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-kube-api-access-jggb7\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.790073 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.789849 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.803678 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.802989 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.803678 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.803591 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.816245 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.816217 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmxd8\" (UniqueName: \"kubernetes.io/projected/3a9a5360-6f89-4b28-9638-a56fdb01389f-kube-api-access-fmxd8\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.816920 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.816899 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.817616 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.817594 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:02.850841 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.829649 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.944790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.925374 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.944790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.927606 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:02.944790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:02.929394 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:03.165196 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:03.165150 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:03.408968 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:03.408525 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz"] Oct 30 20:51:03.411693 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:51:03.411663 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9a5360_6f89_4b28_9638_a56fdb01389f.slice/crio-79fb15329e6e69cd09a8f64b1964e1e064779d7eb2caf99c4533c6c61ee190e8 WatchSource:0}: Error finding container 79fb15329e6e69cd09a8f64b1964e1e064779d7eb2caf99c4533c6c61ee190e8: Status 404 returned error can't find the container with id 79fb15329e6e69cd09a8f64b1964e1e064779d7eb2caf99c4533c6c61ee190e8 Oct 30 20:51:03.411838 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:03.411746 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 20:51:03.411933 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:03.411903 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" podUID="f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 20:51:03.414298 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:03.414264 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:51:03.414392 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:03.414372 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:51:03.414463 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:03.414415 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:51:03.690789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:03.690010 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j"] Oct 30 20:51:03.723484 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:03.722179 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:51:03.723484 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:03.722265 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:51:03.723484 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:03.722303 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:51:04.047185 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:04.046803 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" event={"ID":"3a9a5360-6f89-4b28-9638-a56fdb01389f","Type":"ContainerStarted","Data":"8e520978851e1ebba7ed8eff2dc733057244d6c65c632e1b70d3a5db6b8c7036"} Oct 30 20:51:04.047185 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:04.046865 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" event={"ID":"3a9a5360-6f89-4b28-9638-a56fdb01389f","Type":"ContainerStarted","Data":"79fb15329e6e69cd09a8f64b1964e1e064779d7eb2caf99c4533c6c61ee190e8"} Oct 30 20:51:04.049114 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:04.049091 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" event={"ID":"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b","Type":"ContainerStarted","Data":"6272471aa9cda5bc96709e1a2be742aa7e1b0a63ff6cf268b7c7add707e7f06f"} Oct 30 20:51:04.049244 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:04.049137 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" event={"ID":"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b","Type":"ContainerStarted","Data":"7acac36e150209a5658f26cbd7016990cdfd1272fd11669f90ec62ba9cbde213"} Oct 30 20:51:04.073645 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:04.073580 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" podStartSLOduration=3.0735616119999998 podStartE2EDuration="3.073561612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:04.069383089 +0000 UTC m=+769.249279597" watchObservedRunningTime="2026-10-30 20:51:04.073561612 +0000 UTC m=+769.253458121" Oct 30 20:51:04.101800 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:04.101015 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" podStartSLOduration=2.100998695 podStartE2EDuration="2.100998695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:04.095743201 +0000 UTC m=+769.275639711" watchObservedRunningTime="2026-10-30 20:51:04.100998695 +0000 UTC m=+769.280895204" Oct 30 20:51:04.166888 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:04.166814 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:04.168629 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:04.168360 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:51:04.168629 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:04.168402 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" podUID="a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:51:04.944789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:04.943869 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:51:04.944789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:04.943931 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" podUID="3a9a5360-6f89-4b28-9638-a56fdb01389f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:51:04.955792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:04.949542 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:05.167790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:05.167102 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:51:05.167790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:05.167162 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" podUID="a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:51:05.457789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:05.456879 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:51:05.457789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:05.456943 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" podUID="f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:05.618791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:05.617099 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:51:05.618791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:05.618434 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:51:05.625787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:05.624970 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:51:05.960792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:05.945880 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:51:05.960792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:05.945943 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" podUID="3a9a5360-6f89-4b28-9638-a56fdb01389f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:51:06.166186 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:06.166150 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:51:06.166399 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:06.166207 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" podUID="a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:51:06.932786 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:06.932726 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:51:06.932995 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:06.932802 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" podUID="3a9a5360-6f89-4b28-9638-a56fdb01389f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:51:07.166531 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:07.166498 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:51:07.166947 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:07.166547 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" podUID="a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:51:07.931182 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:07.931142 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:51:07.931460 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:07.931430 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" podUID="3a9a5360-6f89-4b28-9638-a56fdb01389f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:51:08.166288 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:08.166245 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:51:08.166461 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:08.166307 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" podUID="a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:51:08.965000 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:08.938704 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:51:08.965000 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:08.942831 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" podUID="3a9a5360-6f89-4b28-9638-a56fdb01389f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:51:09.157799 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.156049 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs"] Oct 30 20:51:09.165784 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.165235 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.170782 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.169565 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:51:09.170782 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.169630 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" podUID="a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:51:09.177794 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.176092 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs"] Oct 30 20:51:09.198398 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.196133 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.198398 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.196222 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.198398 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.196274 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.198398 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.196316 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/883d31e2-ac6d-466c-900d-d86010b4028e-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.198398 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.196343 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/883d31e2-ac6d-466c-900d-d86010b4028e-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.198398 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.196384 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/883d31e2-ac6d-466c-900d-d86010b4028e-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.198398 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.196410 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.198398 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.196462 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btc44\" (UniqueName: \"kubernetes.io/projected/883d31e2-ac6d-466c-900d-d86010b4028e-kube-api-access-btc44\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.198398 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.196482 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.297206 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.297128 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.297206 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.297171 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.297384 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.297215 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.297384 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.297245 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/883d31e2-ac6d-466c-900d-d86010b4028e-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.297384 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.297271 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/883d31e2-ac6d-466c-900d-d86010b4028e-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.297384 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.297302 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/883d31e2-ac6d-466c-900d-d86010b4028e-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.297384 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.297328 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.297384 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.297369 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btc44\" (UniqueName: \"kubernetes.io/projected/883d31e2-ac6d-466c-900d-d86010b4028e-kube-api-access-btc44\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.297576 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.297390 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.298212 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.298194 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.298457 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.298381 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.298586 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.298472 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.298822 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.298750 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.298924 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.298849 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/883d31e2-ac6d-466c-900d-d86010b4028e-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.300313 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.300288 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/883d31e2-ac6d-466c-900d-d86010b4028e-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.300869 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.300851 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.320480 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.320451 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btc44\" (UniqueName: \"kubernetes.io/projected/883d31e2-ac6d-466c-900d-d86010b4028e-kube-api-access-btc44\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.351788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.336223 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/883d31e2-ac6d-466c-900d-d86010b4028e-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.484974 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.484938 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:09.928787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.915127 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs"] Oct 30 20:51:09.932418 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.932386 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:51:09.932418 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.932450 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" podUID="3a9a5360-6f89-4b28-9638-a56fdb01389f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:51:09.937537 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:51:09.937508 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883d31e2_ac6d_466c_900d_d86010b4028e.slice/crio-75ceebc0f3a4e5dc3ffc72262346231ea76536a4db6e808b1246198708e6b1c8 WatchSource:0}: Error finding container 75ceebc0f3a4e5dc3ffc72262346231ea76536a4db6e808b1246198708e6b1c8: Status 404 returned error can't find the container with id 75ceebc0f3a4e5dc3ffc72262346231ea76536a4db6e808b1246198708e6b1c8 Oct 30 20:51:09.942017 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.941922 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:51:09.942017 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.941998 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:51:09.942193 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:09.942034 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:51:10.158853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:10.158365 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" event={"ID":"883d31e2-ac6d-466c-900d-d86010b4028e","Type":"ContainerStarted","Data":"d96dd712b792441d9bd648e9df77a084aecdf2f904edcc3b3eadc340f3c64956"} Oct 30 20:51:10.158853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:10.158470 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" event={"ID":"883d31e2-ac6d-466c-900d-d86010b4028e","Type":"ContainerStarted","Data":"75ceebc0f3a4e5dc3ffc72262346231ea76536a4db6e808b1246198708e6b1c8"} Oct 30 20:51:10.166127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:10.166095 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 20:51:10.166277 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:10.166168 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" podUID="a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 20:51:10.498785 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:10.487866 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:10.936047 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:10.936011 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 20:51:10.936357 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:10.936308 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" podUID="3a9a5360-6f89-4b28-9638-a56fdb01389f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 20:51:11.489932 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:11.489890 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:51:11.490472 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:11.489969 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" podUID="883d31e2-ac6d-466c-900d-d86010b4028e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 20:51:12.201789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:12.200976 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:51:12.201789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:12.201039 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" podUID="a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:12.489305 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:12.489211 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:51:12.490002 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:12.489581 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" podUID="883d31e2-ac6d-466c-900d-d86010b4028e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 20:51:12.945634 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:12.943840 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:51:12.945634 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:12.943914 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" podUID="3a9a5360-6f89-4b28-9638-a56fdb01389f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:13.214946 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:13.214851 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:51:13.227790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:13.215167 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" podUID="a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:13.488277 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:13.488178 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:51:13.488552 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:13.488520 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" podUID="883d31e2-ac6d-466c-900d-d86010b4028e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 20:51:13.676890 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:13.676842 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:13.945003 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:13.944961 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 20:51:13.967968 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:13.967914 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:13.986911 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:13.986876 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:51:13.987194 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:13.987158 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" podUID="3a9a5360-6f89-4b28-9638-a56fdb01389f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:14.160293 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:14.160245 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:14.196955 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:14.196853 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 20:51:14.277899 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:14.277862 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:51:14.278195 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:14.278162 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" podUID="a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:14.486314 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:14.486191 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:51:14.486586 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:14.486554 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" podUID="883d31e2-ac6d-466c-900d-d86010b4028e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 20:51:14.677894 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:14.677845 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:14.681047 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:14.681021 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:51:14.728095 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:14.727980 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" podStartSLOduration=5.727959079 podStartE2EDuration="5.727959079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:10.197007942 +0000 UTC m=+775.376904451" watchObservedRunningTime="2026-10-30 20:51:14.727959079 +0000 UTC m=+779.907855587" Oct 30 20:51:15.103188 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:15.098505 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:15.104455 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:15.099259 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 20:51:15.167225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:15.144631 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:51:15.167225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:15.145647 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" podUID="3a9a5360-6f89-4b28-9638-a56fdb01389f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:15.242571 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:15.241462 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:51:15.242571 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:15.241557 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:15.242571 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:15.241608 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:51:15.242571 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:15.241643 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:51:15.242571 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:15.241675 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:51:15.242571 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:15.241729 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:15.244663 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:15.244641 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:15.244906 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:15.244894 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:15.248080 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:15.248058 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:51:15.248476 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:15.248454 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:51:15.252937 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:15.252916 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:51:15.486444 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:15.486404 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:51:15.486643 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:15.486469 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" podUID="883d31e2-ac6d-466c-900d-d86010b4028e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 20:51:16.486478 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:16.486444 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:51:16.486986 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:16.486505 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" podUID="883d31e2-ac6d-466c-900d-d86010b4028e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 20:51:17.485637 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:17.485600 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 20:51:17.485818 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:17.485652 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" podUID="883d31e2-ac6d-466c-900d-d86010b4028e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 20:51:19.528329 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:19.528288 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:51:19.528853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:19.528352 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" podUID="883d31e2-ac6d-466c-900d-d86010b4028e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:20.212180 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.212151 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:20.212485 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.212459 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:20.216183 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.216163 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:51:20.369839 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.365295 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp"] Oct 30 20:51:20.403843 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.394310 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" Oct 30 20:51:20.431887 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.429624 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp"] Oct 30 20:51:20.461532 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.461011 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 20:51:20.461532 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.461206 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7d1c765a6192\"" Oct 30 20:51:20.461532 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.461316 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-qn9bd\"" Oct 30 20:51:20.461532 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.461429 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 20:51:20.461532 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.461497 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 20:51:20.466846 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.466751 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e8bd80-f406-40af-adfd-619294e2c3bb-plugins-config-volume\") pod \"payload-processing-external-model-5d5f6f949f-sffgp\" (UID: \"56e8bd80-f406-40af-adfd-619294e2c3bb\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" Oct 30 20:51:20.466992 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.466915 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56e8bd80-f406-40af-adfd-619294e2c3bb-routing-overlay-volume\") pod \"payload-processing-external-model-5d5f6f949f-sffgp\" (UID: \"56e8bd80-f406-40af-adfd-619294e2c3bb\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" Oct 30 20:51:20.467071 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.466995 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/56e8bd80-f406-40af-adfd-619294e2c3bb-provider-credentials\") pod \"payload-processing-external-model-5d5f6f949f-sffgp\" (UID: \"56e8bd80-f406-40af-adfd-619294e2c3bb\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" Oct 30 20:51:20.467133 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.467081 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmcvl\" (UniqueName: \"kubernetes.io/projected/56e8bd80-f406-40af-adfd-619294e2c3bb-kube-api-access-jmcvl\") pod \"payload-processing-external-model-5d5f6f949f-sffgp\" (UID: \"56e8bd80-f406-40af-adfd-619294e2c3bb\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" Oct 30 20:51:20.479887 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.479863 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 20:51:20.576792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.568743 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/56e8bd80-f406-40af-adfd-619294e2c3bb-provider-credentials\") pod \"payload-processing-external-model-5d5f6f949f-sffgp\" (UID: \"56e8bd80-f406-40af-adfd-619294e2c3bb\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" Oct 30 20:51:20.577423 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.577395 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmcvl\" (UniqueName: \"kubernetes.io/projected/56e8bd80-f406-40af-adfd-619294e2c3bb-kube-api-access-jmcvl\") pod \"payload-processing-external-model-5d5f6f949f-sffgp\" (UID: \"56e8bd80-f406-40af-adfd-619294e2c3bb\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" Oct 30 20:51:20.588970 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.578042 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e8bd80-f406-40af-adfd-619294e2c3bb-plugins-config-volume\") pod \"payload-processing-external-model-5d5f6f949f-sffgp\" (UID: \"56e8bd80-f406-40af-adfd-619294e2c3bb\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" Oct 30 20:51:20.589408 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.589377 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56e8bd80-f406-40af-adfd-619294e2c3bb-routing-overlay-volume\") pod \"payload-processing-external-model-5d5f6f949f-sffgp\" (UID: \"56e8bd80-f406-40af-adfd-619294e2c3bb\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" Oct 30 20:51:20.590422 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.588489 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/56e8bd80-f406-40af-adfd-619294e2c3bb-provider-credentials\") pod \"payload-processing-external-model-5d5f6f949f-sffgp\" (UID: \"56e8bd80-f406-40af-adfd-619294e2c3bb\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" Oct 30 20:51:20.590581 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.579475 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e8bd80-f406-40af-adfd-619294e2c3bb-plugins-config-volume\") pod \"payload-processing-external-model-5d5f6f949f-sffgp\" (UID: \"56e8bd80-f406-40af-adfd-619294e2c3bb\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" Oct 30 20:51:20.590690 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.590253 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56e8bd80-f406-40af-adfd-619294e2c3bb-routing-overlay-volume\") pod \"payload-processing-external-model-5d5f6f949f-sffgp\" (UID: \"56e8bd80-f406-40af-adfd-619294e2c3bb\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" Oct 30 20:51:20.612843 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.611583 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmcvl\" (UniqueName: \"kubernetes.io/projected/56e8bd80-f406-40af-adfd-619294e2c3bb-kube-api-access-jmcvl\") pod \"payload-processing-external-model-5d5f6f949f-sffgp\" (UID: \"56e8bd80-f406-40af-adfd-619294e2c3bb\") " pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" Oct 30 20:51:20.782156 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:20.782078 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" Oct 30 20:51:21.077581 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:21.077556 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp"] Oct 30 20:51:21.078449 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:51:21.078415 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e8bd80_f406_40af_adfd_619294e2c3bb.slice/crio-3ba0b774220b5239acad34883efd34c2f01bf1bb2061fc3ecdfb9aa30b886bce WatchSource:0}: Error finding container 3ba0b774220b5239acad34883efd34c2f01bf1bb2061fc3ecdfb9aa30b886bce: Status 404 returned error can't find the container with id 3ba0b774220b5239acad34883efd34c2f01bf1bb2061fc3ecdfb9aa30b886bce Oct 30 20:51:21.464261 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:21.464228 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" event={"ID":"56e8bd80-f406-40af-adfd-619294e2c3bb","Type":"ContainerStarted","Data":"b714dfbff234de194de98464b1aebb10463d504cc1574bc337d9e9a0f9fa4aba"} Oct 30 20:51:21.464435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:21.464273 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" event={"ID":"56e8bd80-f406-40af-adfd-619294e2c3bb","Type":"ContainerStarted","Data":"3ba0b774220b5239acad34883efd34c2f01bf1bb2061fc3ecdfb9aa30b886bce"} Oct 30 20:51:21.464435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:21.464308 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" Oct 30 20:51:21.484933 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:21.484888 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" podStartSLOduration=1.484874063 podStartE2EDuration="1.484874063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:21.480957566 +0000 UTC m=+786.660854086" watchObservedRunningTime="2026-10-30 20:51:21.484874063 +0000 UTC m=+786.664770570" Oct 30 20:51:22.515275 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:51:22.515233 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:51:24.198693 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.198661 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx"] Oct 30 20:51:24.203094 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.203074 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.220552 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.220518 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx"] Oct 30 20:51:24.250484 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.250453 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.250647 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.250491 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.250647 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.250514 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.250647 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.250576 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.250762 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.250689 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.250833 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.250809 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.250875 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.250853 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.250936 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.250921 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7w4t\" (UniqueName: \"kubernetes.io/projected/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-kube-api-access-z7w4t\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.251064 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.251040 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.366026 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.365986 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.367277 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.367068 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.367487 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.367465 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.367748 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.367726 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.367939 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.367913 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.368125 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.368108 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7w4t\" (UniqueName: \"kubernetes.io/projected/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-kube-api-access-z7w4t\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.368348 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.368329 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.368563 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.368546 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.368691 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.368678 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.368841 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.368826 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.370612 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.369858 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.370612 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.370115 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.370612 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.370339 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.372208 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.371701 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.378350 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.378280 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.382398 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.382374 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.405136 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.393325 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.405136 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.393881 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7w4t\" (UniqueName: \"kubernetes.io/projected/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-kube-api-access-z7w4t\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.516697 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.516626 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:24.809538 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.809507 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx"] Oct 30 20:51:24.810813 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:51:24.810782 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a91cf2e_5df1_4d3b_9557_d4b8f826d5a4.slice/crio-910a1e31359dcb8e752608741f450f9916c73dd1692947b5b7f0123b6454bab1 WatchSource:0}: Error finding container 910a1e31359dcb8e752608741f450f9916c73dd1692947b5b7f0123b6454bab1: Status 404 returned error can't find the container with id 910a1e31359dcb8e752608741f450f9916c73dd1692947b5b7f0123b6454bab1 Oct 30 20:51:24.813290 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.813257 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:51:24.813378 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.813322 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:51:24.813378 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:24.813350 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:51:25.506805 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:25.506748 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" event={"ID":"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4","Type":"ContainerStarted","Data":"1fee94e5820b62c606edde78e48dcedf152c59ecae177ace9c6929c5c40509a8"} Oct 30 20:51:25.507296 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:25.506864 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" event={"ID":"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4","Type":"ContainerStarted","Data":"910a1e31359dcb8e752608741f450f9916c73dd1692947b5b7f0123b6454bab1"} Oct 30 20:51:25.517552 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:25.517523 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:25.519107 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:25.519082 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:51:25.519224 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:25.519130 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" podUID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 20:51:25.544245 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:25.544189 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" podStartSLOduration=1.544174452 podStartE2EDuration="1.544174452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:25.534326115 +0000 UTC m=+790.714222634" watchObservedRunningTime="2026-10-30 20:51:25.544174452 +0000 UTC m=+790.724070960" Oct 30 20:51:26.171971 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.171938 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5"] Oct 30 20:51:26.187790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.186202 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.295871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.293782 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5"] Oct 30 20:51:26.386127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.386092 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.386306 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.386198 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.386306 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.386239 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvwgb\" (UniqueName: \"kubernetes.io/projected/96a95230-aa96-4d6b-856e-e29871f3d3bf-kube-api-access-vvwgb\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.386428 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.386367 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.386488 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.386455 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.386545 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.386501 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.386621 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.386587 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.386688 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.386628 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.386749 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.386705 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a95230-aa96-4d6b-856e-e29871f3d3bf-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.487613 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.487581 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a95230-aa96-4d6b-856e-e29871f3d3bf-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.487837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.487623 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.487837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.487676 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.487837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.487697 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvwgb\" (UniqueName: \"kubernetes.io/projected/96a95230-aa96-4d6b-856e-e29871f3d3bf-kube-api-access-vvwgb\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.487837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.487719 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.487837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.487803 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.488122 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.487852 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.488122 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.487938 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.488122 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.487980 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.488320 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.488192 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-workload-certs\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.488320 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.488248 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-data\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.488320 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.488314 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-workload-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.488521 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.488488 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a95230-aa96-4d6b-856e-e29871f3d3bf-istiod-ca-cert\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.488628 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.488569 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-credential-socket\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.490317 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.490291 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-envoy\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.490536 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.490517 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-podinfo\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.497696 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.497669 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-token\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.497844 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.497823 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvwgb\" (UniqueName: \"kubernetes.io/projected/96a95230-aa96-4d6b-856e-e29871f3d3bf-kube-api-access-vvwgb\") pod \"e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.522338 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.522306 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:51:26.525792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.522887 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" podUID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 20:51:26.599056 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.599022 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:26.952829 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.951848 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5"] Oct 30 20:51:26.954633 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.954597 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:51:26.954736 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.954664 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:51:26.954736 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:26.954693 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:51:27.517543 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:27.517097 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:51:27.517543 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:27.517155 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" podUID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 20:51:27.631798 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:27.629873 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" event={"ID":"96a95230-aa96-4d6b-856e-e29871f3d3bf","Type":"ContainerStarted","Data":"ac1fdb670c5c8dfb173984560945b31220d115e913cc25d7cbcc23a47b413f15"} Oct 30 20:51:27.631798 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:27.629930 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" event={"ID":"96a95230-aa96-4d6b-856e-e29871f3d3bf","Type":"ContainerStarted","Data":"c2ac109843844336377c229a6c5fec5e76b11ab85648281354b8a4d35530fc31"} Oct 30 20:51:27.661789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:27.657430 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" podStartSLOduration=1.657409192 podStartE2EDuration="1.657409192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:51:27.656711836 +0000 UTC m=+792.836608343" watchObservedRunningTime="2026-10-30 20:51:27.657409192 +0000 UTC m=+792.837305700" Oct 30 20:51:28.517210 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:28.517173 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:51:28.517557 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:28.517496 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" podUID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 20:51:28.601122 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:28.601089 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:28.607905 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:28.607874 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:51:28.608132 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:28.608105 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" podUID="96a95230-aa96-4d6b-856e-e29871f3d3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 20:51:29.162793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:29.156018 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:29.188577 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:29.186870 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:29.229818 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:29.202950 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:29.232561 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:29.232522 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": context deadline exceeded" Oct 30 20:51:29.256587 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:29.255862 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:29.608791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:29.582886 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:51:29.608791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:29.582969 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" podUID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 20:51:29.634166 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:29.634127 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:51:29.634465 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:29.634430 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" podUID="96a95230-aa96-4d6b-856e-e29871f3d3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 20:51:29.961792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:29.957708 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 20:51:30.163993 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:30.163926 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:30.165674 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:30.165653 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:51:30.233060 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:30.232936 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 20:51:30.249061 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:30.249019 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:51:30.295405 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:30.295358 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 20:51:30.296093 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:30.296069 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:51:30.387403 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:30.387363 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 20:51:30.394329 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:30.389301 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:51:30.607371 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:30.607282 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:51:30.607625 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:30.607589 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" podUID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 20:51:30.637751 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:30.637718 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:51:30.638017 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:30.637987 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" podUID="96a95230-aa96-4d6b-856e-e29871f3d3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 20:51:30.742948 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:30.742909 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:51:30.746799 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:30.743197 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" podUID="8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:30.965817 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:30.964104 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:31.082072 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:31.082041 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:51:31.201827 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:31.186904 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:31.270850 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:31.243840 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 20:51:31.271140 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:31.271105 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Oct 30 20:51:31.301977 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:31.301926 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:51:31.302382 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:31.302344 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" podUID="a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:31.373803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:31.302248 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:31.389907 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:31.389847 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:31.401797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:31.399853 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:51:31.401797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:31.399913 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" podUID="3a9a5360-6f89-4b28-9638-a56fdb01389f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:31.604914 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:31.535188 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:51:31.604914 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:31.535260 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" podUID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 20:51:31.624333 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:31.624297 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:51:31.624644 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:31.624598 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" podUID="96a95230-aa96-4d6b-856e-e29871f3d3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 20:51:31.994791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:31.993915 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 20:51:32.197726 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:32.197668 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:32.262308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:32.261118 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:32.298826 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:32.273854 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:32.327793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:32.326027 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:32.407059 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:32.406910 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-mfvmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:51:32.407059 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:32.406984 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" podUID="0226107b-80ef-4c6e-888c-f89f73108c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:32.429805 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:32.429225 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:32.498962 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:32.498854 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:32.539204 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:32.538498 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:51:32.539204 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:32.538561 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" podUID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 20:51:32.539667 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:32.539551 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:51:32.539667 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:32.539604 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" podUID="16e788c7-69cc-48cb-959a-62c4763c2698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:32.599687 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:32.599647 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:51:32.600011 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:32.599979 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" podUID="96a95230-aa96-4d6b-856e-e29871f3d3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 20:51:32.999070 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:32.999017 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:33.211791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:33.208732 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 20:51:33.303960 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:33.303851 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:33.350792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:33.326287 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 20:51:33.445803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:33.430384 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:33.542242 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:33.542199 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:51:33.542681 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:33.542568 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" podUID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 20:51:33.612977 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:33.602150 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" Oct 30 20:51:33.616266 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:33.615882 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:51:33.616266 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:33.615933 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" podUID="96a95230-aa96-4d6b-856e-e29871f3d3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 20:51:33.751899 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:33.728101 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:51:34.268791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:34.027286 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:34.358797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:34.358109 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 20:51:34.358797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:34.358469 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 20:51:34.645840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:34.608238 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:51:34.645840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:34.608316 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" podUID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 20:51:34.645840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:34.627374 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:51:34.645840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:34.627429 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" podUID="96a95230-aa96-4d6b-856e-e29871f3d3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 20:51:34.704820 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:34.695905 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 20:51:34.735823 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:34.735605 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:35.070955 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:35.070911 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:35.232963 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:35.232920 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 20:51:35.354844 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:35.354631 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp"] Oct 30 20:51:35.382220 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:35.378050 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:35.382220 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:35.379324 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 20:51:35.517836 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:35.517801 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:51:35.518128 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:35.518093 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" podUID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 20:51:35.597844 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:35.597282 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:51:35.606090 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:35.603227 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:51:35.606253 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:35.606094 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:51:35.606253 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:35.606141 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" podUID="96a95230-aa96-4d6b-856e-e29871f3d3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 20:51:35.862148 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:35.861077 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:51:35.862148 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:35.861156 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:51:36.526787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:36.526254 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:51:36.526787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:36.526330 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" podUID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 20:51:36.599408 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:36.599376 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:51:36.599577 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:36.599432 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" podUID="96a95230-aa96-4d6b-856e-e29871f3d3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 20:51:37.517291 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:37.517241 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 20:51:37.517599 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:37.517566 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" podUID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 20:51:37.600895 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:37.600853 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:51:37.601398 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:37.600903 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" podUID="96a95230-aa96-4d6b-856e-e29871f3d3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 20:51:38.599525 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:38.599489 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:51:38.599725 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:38.599555 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" podUID="96a95230-aa96-4d6b-856e-e29871f3d3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 20:51:39.517470 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:39.517418 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:39.560991 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:39.560938 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:51:39.561214 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:39.561017 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" podUID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:39.600167 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:39.600131 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 20:51:39.600339 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:39.600187 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" podUID="96a95230-aa96-4d6b-856e-e29871f3d3bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 20:51:39.880728 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:39.880626 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:39.882635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:39.882607 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:51:40.355986 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.355831 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" podUID="56e8bd80-f406-40af-adfd-619294e2c3bb" containerName="payload-processing" containerID="cri-o://b714dfbff234de194de98464b1aebb10463d504cc1574bc337d9e9a0f9fa4aba" gracePeriod=25 Oct 30 20:51:40.611295 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.609148 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:40.611295 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.610163 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:40.611295 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.610385 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:51:40.638499 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.638473 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" Oct 30 20:51:40.688917 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.688278 4848 generic.go:356] "Generic (PLEG): container finished" podID="56e8bd80-f406-40af-adfd-619294e2c3bb" containerID="b714dfbff234de194de98464b1aebb10463d504cc1574bc337d9e9a0f9fa4aba" exitCode=0 Oct 30 20:51:40.688917 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.688586 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" event={"ID":"56e8bd80-f406-40af-adfd-619294e2c3bb","Type":"ContainerDied","Data":"b714dfbff234de194de98464b1aebb10463d504cc1574bc337d9e9a0f9fa4aba"} Oct 30 20:51:40.688917 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.688695 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" event={"ID":"56e8bd80-f406-40af-adfd-619294e2c3bb","Type":"ContainerDied","Data":"3ba0b774220b5239acad34883efd34c2f01bf1bb2061fc3ecdfb9aa30b886bce"} Oct 30 20:51:40.688917 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.688737 4848 scope.go:122] "RemoveContainer" containerID="b714dfbff234de194de98464b1aebb10463d504cc1574bc337d9e9a0f9fa4aba" Oct 30 20:51:40.688917 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.688857 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56e8bd80-f406-40af-adfd-619294e2c3bb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56e8bd80-f406-40af-adfd-619294e2c3bb-routing-overlay-volume\") pod \"56e8bd80-f406-40af-adfd-619294e2c3bb\" (UID: \"56e8bd80-f406-40af-adfd-619294e2c3bb\") " Oct 30 20:51:40.689704 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.689304 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56e8bd80-f406-40af-adfd-619294e2c3bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e8bd80-f406-40af-adfd-619294e2c3bb-plugins-config-volume\") pod \"56e8bd80-f406-40af-adfd-619294e2c3bb\" (UID: \"56e8bd80-f406-40af-adfd-619294e2c3bb\") " Oct 30 20:51:40.689704 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.689428 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e8bd80-f406-40af-adfd-619294e2c3bb-kube-api-access-jmcvl\" (UniqueName: \"kubernetes.io/projected/56e8bd80-f406-40af-adfd-619294e2c3bb-kube-api-access-jmcvl\") pod \"56e8bd80-f406-40af-adfd-619294e2c3bb\" (UID: \"56e8bd80-f406-40af-adfd-619294e2c3bb\") " Oct 30 20:51:40.689704 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.689497 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e8bd80-f406-40af-adfd-619294e2c3bb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/56e8bd80-f406-40af-adfd-619294e2c3bb-provider-credentials\") pod \"56e8bd80-f406-40af-adfd-619294e2c3bb\" (UID: \"56e8bd80-f406-40af-adfd-619294e2c3bb\") " Oct 30 20:51:40.691084 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.690954 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e8bd80-f406-40af-adfd-619294e2c3bb-plugins-config-volume" pod "56e8bd80-f406-40af-adfd-619294e2c3bb" (UID: "56e8bd80-f406-40af-adfd-619294e2c3bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:51:40.693305 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.690227 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp" Oct 30 20:51:40.701511 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.700124 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e8bd80-f406-40af-adfd-619294e2c3bb-routing-overlay-volume" pod "56e8bd80-f406-40af-adfd-619294e2c3bb" (UID: "56e8bd80-f406-40af-adfd-619294e2c3bb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:51:40.705009 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.704964 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e8bd80-f406-40af-adfd-619294e2c3bb-kube-api-access-jmcvl" pod "56e8bd80-f406-40af-adfd-619294e2c3bb" (UID: "56e8bd80-f406-40af-adfd-619294e2c3bb"). InnerVolumeSpecName "kube-api-access-jmcvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:40.706805 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.706322 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e8bd80-f406-40af-adfd-619294e2c3bb-provider-credentials" pod "56e8bd80-f406-40af-adfd-619294e2c3bb" (UID: "56e8bd80-f406-40af-adfd-619294e2c3bb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:51:40.743235 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.743211 4848 scope.go:122] "RemoveContainer" containerID="b714dfbff234de194de98464b1aebb10463d504cc1574bc337d9e9a0f9fa4aba" Oct 30 20:51:40.743612 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:51:40.743587 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b714dfbff234de194de98464b1aebb10463d504cc1574bc337d9e9a0f9fa4aba\": container with ID starting with b714dfbff234de194de98464b1aebb10463d504cc1574bc337d9e9a0f9fa4aba not found: ID does not exist" containerID="b714dfbff234de194de98464b1aebb10463d504cc1574bc337d9e9a0f9fa4aba" Oct 30 20:51:40.743721 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.743624 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b714dfbff234de194de98464b1aebb10463d504cc1574bc337d9e9a0f9fa4aba"} err="failed to get container status \"b714dfbff234de194de98464b1aebb10463d504cc1574bc337d9e9a0f9fa4aba\": rpc error: code = NotFound desc = could not find container \"b714dfbff234de194de98464b1aebb10463d504cc1574bc337d9e9a0f9fa4aba\": container with ID starting with b714dfbff234de194de98464b1aebb10463d504cc1574bc337d9e9a0f9fa4aba not found: ID does not exist" Oct 30 20:51:40.801564 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.795747 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e8bd80-f406-40af-adfd-619294e2c3bb-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:40.801564 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.795803 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmcvl\" (UniqueName: \"kubernetes.io/projected/56e8bd80-f406-40af-adfd-619294e2c3bb-kube-api-access-jmcvl\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:40.801564 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.795830 4848 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/56e8bd80-f406-40af-adfd-619294e2c3bb-provider-credentials\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:40.801564 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:40.795845 4848 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/56e8bd80-f406-40af-adfd-619294e2c3bb-routing-overlay-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:51:41.011858 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:41.011831 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp"] Oct 30 20:51:41.018205 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:41.018180 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f6f949f-sffgp"] Oct 30 20:51:42.971608 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:42.971569 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56e8bd80-f406-40af-adfd-619294e2c3bb" path="/var/lib/kubelet/pods/56e8bd80-f406-40af-adfd-619294e2c3bb/volumes" Oct 30 20:51:45.996974 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:45.996909 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:46.190793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:46.190031 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:46.290309 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:46.288969 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:51:46.290309 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:46.289096 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" podUID="3a9a5360-6f89-4b28-9638-a56fdb01389f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:46.721792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:46.721155 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:47.040849 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:47.040064 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:47.086800 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:47.048296 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:51:47.227012 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:47.226944 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:47.263100 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:47.263068 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:51:47.391866 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:47.292061 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:47.417803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:47.393019 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 20:51:47.494948 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:47.494914 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:51:47.495221 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:47.495190 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" podUID="16e788c7-69cc-48cb-959a-62c4763c2698" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:47.735787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:47.722857 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 20:51:47.739788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:47.737869 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:51:48.206919 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:48.206864 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:48.385793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:48.382033 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:48.435795 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:48.432387 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:48.435795 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:48.432918 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:51:48.544841 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:48.544313 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:48.622797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:48.588439 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:51:48.744962 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:48.744913 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:48.892685 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:48.891732 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:51:49.118027 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:49.117992 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:51:49.223178 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:49.222294 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:49.292945 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:49.292880 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:51:49.418862 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:49.418712 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 20:51:49.597517 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:49.597125 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:51:49.662674 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:49.662476 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:51:49.666746 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:51:49.666721 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:51:52.511100 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:51:52.511057 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:52:12.515791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:12.514850 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:12.552383 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:12.552332 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:12.557790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:12.555875 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:13.087792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.087334 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 20:52:13.113789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.110008 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:13.113789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.113537 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m"] Oct 30 20:52:13.261791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.213529 4848 cpu_manager.go:414] "containerMap: removing container" podUID="56e8bd80-f406-40af-adfd-619294e2c3bb" containerName="payload-processing" Oct 30 20:52:13.286849 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.286812 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56e8bd80-f406-40af-adfd-619294e2c3bb" containerName="payload-processing" Oct 30 20:52:13.318799 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.287324 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="56e8bd80-f406-40af-adfd-619294e2c3bb" containerName="payload-processing" Oct 30 20:52:13.501054 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.501012 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m"] Oct 30 20:52:13.501422 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.501401 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.505902 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.505870 4848 patch_prober.go:29] interesting pod/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:52:13.506131 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.506101 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" podUID="e57add2c-5ccf-47eb-8adb-d5852289b111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:13.532585 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.524844 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:13.533958 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.533933 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:52:13.572790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.569858 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:13.573709 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.573678 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:52:13.641788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.637050 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea433a7-4828-4946-9086-3976cea6c973-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.641788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.637133 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.641788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.637207 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgj9n\" (UniqueName: \"kubernetes.io/projected/cea433a7-4828-4946-9086-3976cea6c973-kube-api-access-qgj9n\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.641788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.637254 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea433a7-4828-4946-9086-3976cea6c973-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.641788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.637314 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.641788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.637360 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.641788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.637398 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.641788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.637611 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea433a7-4828-4946-9086-3976cea6c973-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.641788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.637653 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.804629 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.743543 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea433a7-4828-4946-9086-3976cea6c973-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.804629 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.743615 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.804629 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.743664 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.804629 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.743705 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.804629 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.743910 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea433a7-4828-4946-9086-3976cea6c973-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.804629 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.743952 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.804629 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.744010 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea433a7-4828-4946-9086-3976cea6c973-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.804629 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.744065 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.804629 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.744133 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgj9n\" (UniqueName: \"kubernetes.io/projected/cea433a7-4828-4946-9086-3976cea6c973-kube-api-access-qgj9n\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.852908 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.852860 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:13.886796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.873679 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.886796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.873981 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.963827 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.892908 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.963827 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.900247 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea433a7-4828-4946-9086-3976cea6c973-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:13.963827 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:13.917567 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:14.012791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:14.009872 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea433a7-4828-4946-9086-3976cea6c973-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:14.124501 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:14.100986 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:14.124501 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:14.101578 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:52:14.124501 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:14.111853 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 20:52:14.124501 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:14.112151 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:52:14.205793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:14.194103 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:14.282206 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:14.282170 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgj9n\" (UniqueName: \"kubernetes.io/projected/cea433a7-4828-4946-9086-3976cea6c973-kube-api-access-qgj9n\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:14.344195 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:14.319410 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea433a7-4828-4946-9086-3976cea6c973-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:14.431851 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:14.414135 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:14.543793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:14.517553 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:52:14.581652 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:14.571909 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:14.796172 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:14.796145 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:52:14.797827 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:14.797799 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:52:14.921790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:14.870250 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 20:52:14.921790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:14.870943 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:52:15.181297 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:15.181269 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:52:15.181570 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:15.181542 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:52:15.971966 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:15.971928 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m"] Oct 30 20:52:15.974842 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:15.974805 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:15.974976 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:15.974887 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:15.974976 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:15.974931 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:16.717796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:16.717750 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" event={"ID":"cea433a7-4828-4946-9086-3976cea6c973","Type":"ContainerStarted","Data":"4d853b05930415906f00b32301334f471967b1033e6c61b98ac75098c8ab6a1f"} Oct 30 20:52:16.718295 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:16.717814 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" event={"ID":"cea433a7-4828-4946-9086-3976cea6c973","Type":"ContainerStarted","Data":"55d567324e000ab87b69ada9ad9e6d36a0cd25651342a9f245de586d42efa297"} Oct 30 20:52:16.743177 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:16.743126 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" podStartSLOduration=3.743112607 podStartE2EDuration="3.743112607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:16.74063489 +0000 UTC m=+841.920531398" watchObservedRunningTime="2026-10-30 20:52:16.743112607 +0000 UTC m=+841.923009113" Oct 30 20:52:17.433866 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:17.433275 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:17.434919 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:17.434885 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:52:17.435056 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:17.434954 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" podUID="cea433a7-4828-4946-9086-3976cea6c973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 20:52:18.433237 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:18.433202 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:52:18.433421 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:18.433255 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" podUID="cea433a7-4828-4946-9086-3976cea6c973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 20:52:19.433293 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:19.433254 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:52:19.433715 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:19.433313 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" podUID="cea433a7-4828-4946-9086-3976cea6c973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 20:52:20.433364 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:20.433325 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:52:20.433870 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:20.433385 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" podUID="cea433a7-4828-4946-9086-3976cea6c973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 20:52:21.435260 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:21.435224 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:52:21.449210 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:21.449164 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" podUID="cea433a7-4828-4946-9086-3976cea6c973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 20:52:22.432644 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:22.432609 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 20:52:22.432943 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:22.432915 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" podUID="cea433a7-4828-4946-9086-3976cea6c973" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 20:52:22.530171 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:52:22.530132 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:52:23.581868 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.581828 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl"] Oct 30 20:52:23.629028 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.628995 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl"] Oct 30 20:52:23.644046 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.630964 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.666232 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.666200 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1fac16-openshift-default-dockercfg-vvhxw\"" Oct 30 20:52:23.805308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.790651 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-token\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.805308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.790711 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4bgw\" (UniqueName: \"kubernetes.io/projected/a79a6860-1766-4d6b-9bb7-ccfa69a02989-kube-api-access-k4bgw\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.805308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.790752 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-envoy\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.873857 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.855001 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-workload-certs\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.873857 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.855149 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-workload-socket\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.873857 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.855260 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istiod-ca-cert\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.873857 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.855295 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-credential-socket\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.873857 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.855329 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-data\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.873857 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.855411 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-podinfo\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.956379 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.956339 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-token\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.956379 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.956385 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4bgw\" (UniqueName: \"kubernetes.io/projected/a79a6860-1766-4d6b-9bb7-ccfa69a02989-kube-api-access-k4bgw\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.956599 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.956500 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-envoy\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.956647 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.956599 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-workload-certs\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.956709 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.956687 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-workload-socket\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.956813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.956785 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istiod-ca-cert\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.956890 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.956819 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-credential-socket\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.956890 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.956841 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-data\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.957056 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.957028 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-workload-certs\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.957222 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.957115 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-podinfo\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.957343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.957251 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-data\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.957460 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.957347 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-workload-socket\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.957549 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.957510 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-credential-socket\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.957862 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.957837 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istiod-ca-cert\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.959722 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.959699 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-envoy\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.960015 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.959994 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-podinfo\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.966739 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.966673 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-token\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:23.966976 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:23.966952 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4bgw\" (UniqueName: \"kubernetes.io/projected/a79a6860-1766-4d6b-9bb7-ccfa69a02989-kube-api-access-k4bgw\") pod \"e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:24.041049 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:24.041021 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:24.041413 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:24.041391 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:24.042266 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:24.042249 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 20:52:24.212153 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:24.212116 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:24.514029 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:24.513946 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl"] Oct 30 20:52:24.517718 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:52:24.517685 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79a6860_1766_4d6b_9bb7_ccfa69a02989.slice/crio-9f1c46e99a83c43b0d659268a92d509ba0dab83f6db27dc1b78640e2f6da3742 WatchSource:0}: Error finding container 9f1c46e99a83c43b0d659268a92d509ba0dab83f6db27dc1b78640e2f6da3742: Status 404 returned error can't find the container with id 9f1c46e99a83c43b0d659268a92d509ba0dab83f6db27dc1b78640e2f6da3742 Oct 30 20:52:24.520407 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:24.520366 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:24.520505 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:24.520458 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:24.520548 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:24.520502 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:25.041803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:25.035629 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" event={"ID":"a79a6860-1766-4d6b-9bb7-ccfa69a02989","Type":"ContainerStarted","Data":"12f5e8040c139384dd8b5f1fea17ac812928b78f1d108401eca7208ac2caba64"} Oct 30 20:52:25.041803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:25.035700 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" event={"ID":"a79a6860-1766-4d6b-9bb7-ccfa69a02989","Type":"ContainerStarted","Data":"9f1c46e99a83c43b0d659268a92d509ba0dab83f6db27dc1b78640e2f6da3742"} Oct 30 20:52:25.050791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:25.049584 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" podStartSLOduration=2.049565686 podStartE2EDuration="2.049565686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:25.046837462 +0000 UTC m=+850.226733970" watchObservedRunningTime="2026-10-30 20:52:25.049565686 +0000 UTC m=+850.229462194" Oct 30 20:52:25.215417 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:25.215380 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:25.222249 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:25.222044 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:26.001716 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.001687 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:26.002750 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.002727 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:52:26.244866 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.244826 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8"] Oct 30 20:52:26.249267 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.249236 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8" Oct 30 20:52:26.253162 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.253078 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5wv5\" (UniqueName: \"kubernetes.io/projected/e029fa76-d526-498e-acbd-efda2b093887-kube-api-access-d5wv5\") pod \"payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8\" (UID: \"e029fa76-d526-498e-acbd-efda2b093887\") " pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8" Oct 30 20:52:26.253298 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.253184 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e029fa76-d526-498e-acbd-efda2b093887-plugins-config-volume\") pod \"payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8\" (UID: \"e029fa76-d526-498e-acbd-efda2b093887\") " pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8" Oct 30 20:52:26.260555 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.260523 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8"] Oct 30 20:52:26.267960 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.267931 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a52bd2a1\"" Oct 30 20:52:26.267960 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.267958 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a52bd2a1-dockercfg-25j5w\"" Oct 30 20:52:26.284648 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.284622 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj"] Oct 30 20:52:26.292837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.290622 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj" Oct 30 20:52:26.313071 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.313040 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj"] Oct 30 20:52:26.370287 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.357711 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f2d488-7956-4a7a-8b44-53eabadd98bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj\" (UID: \"c3f2d488-7956-4a7a-8b44-53eabadd98bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj" Oct 30 20:52:26.370287 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.357983 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdlrp\" (UniqueName: \"kubernetes.io/projected/c3f2d488-7956-4a7a-8b44-53eabadd98bc-kube-api-access-bdlrp\") pod \"payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj\" (UID: \"c3f2d488-7956-4a7a-8b44-53eabadd98bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj" Oct 30 20:52:26.370287 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.358133 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5wv5\" (UniqueName: \"kubernetes.io/projected/e029fa76-d526-498e-acbd-efda2b093887-kube-api-access-d5wv5\") pod \"payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8\" (UID: \"e029fa76-d526-498e-acbd-efda2b093887\") " pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8" Oct 30 20:52:26.370287 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.358188 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e029fa76-d526-498e-acbd-efda2b093887-plugins-config-volume\") pod \"payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8\" (UID: \"e029fa76-d526-498e-acbd-efda2b093887\") " pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8" Oct 30 20:52:26.370287 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.359644 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e029fa76-d526-498e-acbd-efda2b093887-plugins-config-volume\") pod \"payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8\" (UID: \"e029fa76-d526-498e-acbd-efda2b093887\") " pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8" Oct 30 20:52:26.411788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.407936 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5wv5\" (UniqueName: \"kubernetes.io/projected/e029fa76-d526-498e-acbd-efda2b093887-kube-api-access-d5wv5\") pod \"payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8\" (UID: \"e029fa76-d526-498e-acbd-efda2b093887\") " pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8" Oct 30 20:52:26.465706 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.463780 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f2d488-7956-4a7a-8b44-53eabadd98bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj\" (UID: \"c3f2d488-7956-4a7a-8b44-53eabadd98bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj" Oct 30 20:52:26.465706 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.463931 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdlrp\" (UniqueName: \"kubernetes.io/projected/c3f2d488-7956-4a7a-8b44-53eabadd98bc-kube-api-access-bdlrp\") pod \"payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj\" (UID: \"c3f2d488-7956-4a7a-8b44-53eabadd98bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj" Oct 30 20:52:26.465706 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.465164 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f2d488-7956-4a7a-8b44-53eabadd98bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj\" (UID: \"c3f2d488-7956-4a7a-8b44-53eabadd98bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj" Oct 30 20:52:26.489867 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.489800 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdlrp\" (UniqueName: \"kubernetes.io/projected/c3f2d488-7956-4a7a-8b44-53eabadd98bc-kube-api-access-bdlrp\") pod \"payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj\" (UID: \"c3f2d488-7956-4a7a-8b44-53eabadd98bc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj" Oct 30 20:52:26.567012 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.566501 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8" Oct 30 20:52:26.616979 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.616601 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj" Oct 30 20:52:26.885354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.885326 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8"] Oct 30 20:52:26.886190 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:52:26.886165 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode029fa76_d526_498e_acbd_efda2b093887.slice/crio-c8e87100442cfdd6eba885ecfa71df182f16d45a8e25eaa9660314bb31b74e7b WatchSource:0}: Error finding container c8e87100442cfdd6eba885ecfa71df182f16d45a8e25eaa9660314bb31b74e7b: Status 404 returned error can't find the container with id c8e87100442cfdd6eba885ecfa71df182f16d45a8e25eaa9660314bb31b74e7b Oct 30 20:52:26.915733 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:26.915708 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj"] Oct 30 20:52:26.919021 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:52:26.918987 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f2d488_7956_4a7a_8b44_53eabadd98bc.slice/crio-7fa062dad33ac66392b86786d8600db71513e0234c066a6bd77aae33c4b220fc WatchSource:0}: Error finding container 7fa062dad33ac66392b86786d8600db71513e0234c066a6bd77aae33c4b220fc: Status 404 returned error can't find the container with id 7fa062dad33ac66392b86786d8600db71513e0234c066a6bd77aae33c4b220fc Oct 30 20:52:27.009498 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:27.009466 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj" event={"ID":"c3f2d488-7956-4a7a-8b44-53eabadd98bc","Type":"ContainerStarted","Data":"a6caef4dce9bdad6e10a6bc0f6a25afab8d63beafab4412349881c8074b4296d"} Oct 30 20:52:27.009688 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:27.009522 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj" event={"ID":"c3f2d488-7956-4a7a-8b44-53eabadd98bc","Type":"ContainerStarted","Data":"7fa062dad33ac66392b86786d8600db71513e0234c066a6bd77aae33c4b220fc"} Oct 30 20:52:27.009688 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:27.009601 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj" Oct 30 20:52:27.011746 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:27.011720 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8" event={"ID":"e029fa76-d526-498e-acbd-efda2b093887","Type":"ContainerStarted","Data":"8823367d236c4ea9dfa1cf69adb3bf1886f17c4b8d7fa6e68ca4b343e7bde1a5"} Oct 30 20:52:27.011896 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:27.011797 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8" event={"ID":"e029fa76-d526-498e-acbd-efda2b093887","Type":"ContainerStarted","Data":"c8e87100442cfdd6eba885ecfa71df182f16d45a8e25eaa9660314bb31b74e7b"} Oct 30 20:52:27.011896 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:27.011831 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8" Oct 30 20:52:27.032547 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:27.032499 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj" podStartSLOduration=1.032487756 podStartE2EDuration="1.032487756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:27.030843201 +0000 UTC m=+852.210739709" watchObservedRunningTime="2026-10-30 20:52:27.032487756 +0000 UTC m=+852.212384302" Oct 30 20:52:27.052846 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:27.052797 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8" podStartSLOduration=1.052760414 podStartE2EDuration="1.052760414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:27.049097277 +0000 UTC m=+852.228993784" watchObservedRunningTime="2026-10-30 20:52:27.052760414 +0000 UTC m=+852.232656944" Oct 30 20:52:28.949080 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:28.949042 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28"] Oct 30 20:52:28.953457 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:28.953432 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28" Oct 30 20:52:28.977139 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:28.977111 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28"] Oct 30 20:52:29.110790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:29.072505 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1fac16\"" Oct 30 20:52:29.111039 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:29.080060 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1fac16-dockercfg-9xxdq\"" Oct 30 20:52:29.155791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:29.153669 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6e17b1-e385-470d-89f0-25397789a650-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28\" (UID: \"2c6e17b1-e385-470d-89f0-25397789a650\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28" Oct 30 20:52:29.155791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:29.153885 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7hjd\" (UniqueName: \"kubernetes.io/projected/2c6e17b1-e385-470d-89f0-25397789a650-kube-api-access-m7hjd\") pod \"payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28\" (UID: \"2c6e17b1-e385-470d-89f0-25397789a650\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28" Oct 30 20:52:29.259837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:29.256823 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6e17b1-e385-470d-89f0-25397789a650-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28\" (UID: \"2c6e17b1-e385-470d-89f0-25397789a650\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28" Oct 30 20:52:29.259837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:29.257021 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7hjd\" (UniqueName: \"kubernetes.io/projected/2c6e17b1-e385-470d-89f0-25397789a650-kube-api-access-m7hjd\") pod \"payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28\" (UID: \"2c6e17b1-e385-470d-89f0-25397789a650\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28" Oct 30 20:52:29.259837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:29.258275 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6e17b1-e385-470d-89f0-25397789a650-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28\" (UID: \"2c6e17b1-e385-470d-89f0-25397789a650\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28" Oct 30 20:52:29.292817 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:29.291520 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7hjd\" (UniqueName: \"kubernetes.io/projected/2c6e17b1-e385-470d-89f0-25397789a650-kube-api-access-m7hjd\") pod \"payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28\" (UID: \"2c6e17b1-e385-470d-89f0-25397789a650\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28" Oct 30 20:52:29.580836 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:29.576169 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28" Oct 30 20:52:29.865994 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:29.865966 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28"] Oct 30 20:52:29.866971 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:52:29.866945 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c6e17b1_e385_470d_89f0_25397789a650.slice/crio-ec6b4354df17a841225013da067600248ecf2e397920064d28e1455c2e18a3d1 WatchSource:0}: Error finding container ec6b4354df17a841225013da067600248ecf2e397920064d28e1455c2e18a3d1: Status 404 returned error can't find the container with id ec6b4354df17a841225013da067600248ecf2e397920064d28e1455c2e18a3d1 Oct 30 20:52:30.215957 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:30.215918 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28" event={"ID":"2c6e17b1-e385-470d-89f0-25397789a650","Type":"ContainerStarted","Data":"5cc24cc74784ac6d48b40388cb712779c76b86db237e92629044974f873ab3c5"} Oct 30 20:52:30.216544 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:30.216518 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28" event={"ID":"2c6e17b1-e385-470d-89f0-25397789a650","Type":"ContainerStarted","Data":"ec6b4354df17a841225013da067600248ecf2e397920064d28e1455c2e18a3d1"} Oct 30 20:52:30.217057 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:30.217015 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28" Oct 30 20:52:30.255735 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:30.255675 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28" podStartSLOduration=2.255656213 podStartE2EDuration="2.255656213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:30.253663886 +0000 UTC m=+855.433560394" watchObservedRunningTime="2026-10-30 20:52:30.255656213 +0000 UTC m=+855.435552723" Oct 30 20:52:32.466864 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:32.466818 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw"] Oct 30 20:52:32.485029 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:32.484997 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw"] Oct 30 20:52:32.485384 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:32.485363 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" Oct 30 20:52:32.506790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:32.494059 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a52bd2a1-serving-cert\"" Oct 30 20:52:32.543137 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:32.543094 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78c510b-f4e1-4d65-93cc-2b00f0896b81-openshift-service-ca\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw\" (UID: \"e78c510b-f4e1-4d65-93cc-2b00f0896b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" Oct 30 20:52:32.544167 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:32.544144 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e78c510b-f4e1-4d65-93cc-2b00f0896b81-maas-api-tls\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw\" (UID: \"e78c510b-f4e1-4d65-93cc-2b00f0896b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" Oct 30 20:52:32.544317 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:32.544303 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e78c510b-f4e1-4d65-93cc-2b00f0896b81-metrics-certs\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw\" (UID: \"e78c510b-f4e1-4d65-93cc-2b00f0896b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" Oct 30 20:52:32.544465 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:32.544452 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qtjj\" (UniqueName: \"kubernetes.io/projected/e78c510b-f4e1-4d65-93cc-2b00f0896b81-kube-api-access-7qtjj\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw\" (UID: \"e78c510b-f4e1-4d65-93cc-2b00f0896b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" Oct 30 20:52:32.645789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:32.645603 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78c510b-f4e1-4d65-93cc-2b00f0896b81-openshift-service-ca\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw\" (UID: \"e78c510b-f4e1-4d65-93cc-2b00f0896b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" Oct 30 20:52:32.645789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:32.645695 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e78c510b-f4e1-4d65-93cc-2b00f0896b81-maas-api-tls\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw\" (UID: \"e78c510b-f4e1-4d65-93cc-2b00f0896b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" Oct 30 20:52:32.645789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:32.645726 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e78c510b-f4e1-4d65-93cc-2b00f0896b81-metrics-certs\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw\" (UID: \"e78c510b-f4e1-4d65-93cc-2b00f0896b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" Oct 30 20:52:32.648108 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:32.647423 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78c510b-f4e1-4d65-93cc-2b00f0896b81-openshift-service-ca\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw\" (UID: \"e78c510b-f4e1-4d65-93cc-2b00f0896b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" Oct 30 20:52:32.648108 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:32.647538 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qtjj\" (UniqueName: \"kubernetes.io/projected/e78c510b-f4e1-4d65-93cc-2b00f0896b81-kube-api-access-7qtjj\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw\" (UID: \"e78c510b-f4e1-4d65-93cc-2b00f0896b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" Oct 30 20:52:32.657799 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:32.654116 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e78c510b-f4e1-4d65-93cc-2b00f0896b81-maas-api-tls\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw\" (UID: \"e78c510b-f4e1-4d65-93cc-2b00f0896b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" Oct 30 20:52:32.684798 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:32.682379 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e78c510b-f4e1-4d65-93cc-2b00f0896b81-metrics-certs\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw\" (UID: \"e78c510b-f4e1-4d65-93cc-2b00f0896b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" Oct 30 20:52:32.709790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:32.706486 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qtjj\" (UniqueName: \"kubernetes.io/projected/e78c510b-f4e1-4d65-93cc-2b00f0896b81-kube-api-access-7qtjj\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw\" (UID: \"e78c510b-f4e1-4d65-93cc-2b00f0896b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" Oct 30 20:52:32.844697 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:32.843253 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" Oct 30 20:52:33.875206 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:33.872348 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:33.969792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:33.805222 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:34.021366 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:34.021084 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:34.122983 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:34.122903 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:34.679489 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:34.677886 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Oct 30 20:52:34.679489 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:34.677853 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 20:52:34.907795 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:34.876133 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:34.907795 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:34.898867 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:52:35.046792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:34.955126 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:35.075792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:35.057851 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 20:52:35.128697 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:35.128655 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 20:52:35.430849 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:35.419067 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:52:35.430849 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:35.419133 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:52:35.430849 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:35.419171 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:52:35.706187 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:35.698013 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 20:52:35.706187 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:35.698609 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:52:35.977851 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:35.977646 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 20:52:35.988789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:35.980554 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 20:52:36.021793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:35.996509 4848 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:52:36.087790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:36.069461 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 20:52:36.581793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:36.358274 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:36.581793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:36.358371 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw"] Oct 30 20:52:36.581793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:36.395962 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:52:36.581793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:36.477577 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" podUID="883d31e2-ac6d-466c-900d-d86010b4028e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:36.581793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:36.430083 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:52:36.581793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:36.477898 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" podUID="9829d8aa-db34-42b9-bf27-316fa4cc86f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:36.744791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:36.734557 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:37.000851 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:36.992941 4848 patch_prober.go:29] interesting pod/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:52:37.000851 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:36.993007 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" podUID="f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:37.076999 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:37.076942 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 20:52:37.200102 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:37.199985 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 20:52:37.258846 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:37.237876 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 20:52:37.485978 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:37.485920 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:37.748815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:37.744578 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:37.834207 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:37.834120 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:52:37.835881 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:37.834831 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:52:37.836789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:37.836087 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:52:37.837067 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:37.837051 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:52:37.870746 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:37.870718 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:52:37.981037 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:37.969647 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" event={"ID":"e78c510b-f4e1-4d65-93cc-2b00f0896b81","Type":"ContainerStarted","Data":"cd3d67682e3ecbdb4fab99656022bb358d8d15a7e7aec4e98862d79f274e1307"} Oct 30 20:52:38.026797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:38.024260 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj" Oct 30 20:52:38.026797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:38.025507 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8" Oct 30 20:52:38.977754 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:38.977716 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" event={"ID":"e78c510b-f4e1-4d65-93cc-2b00f0896b81","Type":"ContainerStarted","Data":"92b953df46ce805af56d91f8103d5d0f9e08d386e2985c54d67fa6b404ef87fd"} Oct 30 20:52:38.977948 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:38.977796 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" Oct 30 20:52:39.004783 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:39.004728 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" podStartSLOduration=5.114372849 podStartE2EDuration="7.004713631s" totalImagesPullingTime="1.890340782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:39.002430483 +0000 UTC m=+864.182327002" watchObservedRunningTime="2026-10-30 20:52:39.004713631 +0000 UTC m=+864.184610138" Oct 30 20:52:39.085796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:39.085751 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb"] Oct 30 20:52:39.090712 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:39.090689 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" Oct 30 20:52:39.102547 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:39.102511 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb"] Oct 30 20:52:39.104637 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:39.103992 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1fac16-serving-cert\"" Oct 30 20:52:39.107511 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:39.107485 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706abb91-d2cd-4f79-8247-2f9077a602c6-maas-api-tls\") pod \"maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb\" (UID: \"706abb91-d2cd-4f79-8247-2f9077a602c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" Oct 30 20:52:39.107819 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:39.107781 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/706abb91-d2cd-4f79-8247-2f9077a602c6-openshift-service-ca\") pod \"maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb\" (UID: \"706abb91-d2cd-4f79-8247-2f9077a602c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" Oct 30 20:52:39.107942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:39.107876 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/706abb91-d2cd-4f79-8247-2f9077a602c6-metrics-certs\") pod \"maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb\" (UID: \"706abb91-d2cd-4f79-8247-2f9077a602c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" Oct 30 20:52:39.107942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:39.107908 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmcm9\" (UniqueName: \"kubernetes.io/projected/706abb91-d2cd-4f79-8247-2f9077a602c6-kube-api-access-bmcm9\") pod \"maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb\" (UID: \"706abb91-d2cd-4f79-8247-2f9077a602c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" Oct 30 20:52:39.216402 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:39.216365 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706abb91-d2cd-4f79-8247-2f9077a602c6-maas-api-tls\") pod \"maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb\" (UID: \"706abb91-d2cd-4f79-8247-2f9077a602c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" Oct 30 20:52:39.216685 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:39.216570 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/706abb91-d2cd-4f79-8247-2f9077a602c6-openshift-service-ca\") pod \"maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb\" (UID: \"706abb91-d2cd-4f79-8247-2f9077a602c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" Oct 30 20:52:39.216685 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:39.216615 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/706abb91-d2cd-4f79-8247-2f9077a602c6-metrics-certs\") pod \"maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb\" (UID: \"706abb91-d2cd-4f79-8247-2f9077a602c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" Oct 30 20:52:39.216685 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:39.216646 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmcm9\" (UniqueName: \"kubernetes.io/projected/706abb91-d2cd-4f79-8247-2f9077a602c6-kube-api-access-bmcm9\") pod \"maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb\" (UID: \"706abb91-d2cd-4f79-8247-2f9077a602c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" Oct 30 20:52:39.217717 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:52:39.217246 4848 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-serving-cert: secret "maas-api-e2e-api-a-1fac16-serving-cert" not found Oct 30 20:52:39.217717 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:52:39.217317 4848 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/706abb91-d2cd-4f79-8247-2f9077a602c6-maas-api-tls podName:706abb91-d2cd-4f79-8247-2f9077a602c6 nodeName:}" failed. No retries permitted until 2026-10-30 20:52:39.717293937 +0000 UTC m=+864.897190452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/706abb91-d2cd-4f79-8247-2f9077a602c6-maas-api-tls") pod "maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" (UID: "706abb91-d2cd-4f79-8247-2f9077a602c6") : secret "maas-api-e2e-api-a-1fac16-serving-cert" not found Oct 30 20:52:39.218221 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:39.218179 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/706abb91-d2cd-4f79-8247-2f9077a602c6-openshift-service-ca\") pod \"maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb\" (UID: \"706abb91-d2cd-4f79-8247-2f9077a602c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" Oct 30 20:52:39.273835 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:39.270038 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/706abb91-d2cd-4f79-8247-2f9077a602c6-metrics-certs\") pod \"maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb\" (UID: \"706abb91-d2cd-4f79-8247-2f9077a602c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" Oct 30 20:52:39.273835 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:39.271093 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmcm9\" (UniqueName: \"kubernetes.io/projected/706abb91-d2cd-4f79-8247-2f9077a602c6-kube-api-access-bmcm9\") pod \"maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb\" (UID: \"706abb91-d2cd-4f79-8247-2f9077a602c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" Oct 30 20:52:39.724160 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:39.724125 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706abb91-d2cd-4f79-8247-2f9077a602c6-maas-api-tls\") pod \"maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb\" (UID: \"706abb91-d2cd-4f79-8247-2f9077a602c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" Oct 30 20:52:39.726537 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:39.726513 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706abb91-d2cd-4f79-8247-2f9077a602c6-maas-api-tls\") pod \"maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb\" (UID: \"706abb91-d2cd-4f79-8247-2f9077a602c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" Oct 30 20:52:40.063386 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:40.063289 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" Oct 30 20:52:40.395628 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:40.395243 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb"] Oct 30 20:52:41.067968 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:41.067884 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" event={"ID":"706abb91-d2cd-4f79-8247-2f9077a602c6","Type":"ContainerStarted","Data":"58a60bdfad2954ce7a1763b7544f78ffae9cd88796159667a26f7cd0d8d6c3eb"} Oct 30 20:52:41.067968 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:41.067934 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" event={"ID":"706abb91-d2cd-4f79-8247-2f9077a602c6","Type":"ContainerStarted","Data":"7fd5f75fc441a972071eee1df42833aaf89cc0440db3f25206a6a9318d6e2f3a"} Oct 30 20:52:41.068213 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:41.068045 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" Oct 30 20:52:41.087359 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:41.087303 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" podStartSLOduration=1.695853732 podStartE2EDuration="2.087287037s" totalImagesPullingTime="391.433305ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:41.084890426 +0000 UTC m=+866.264786941" watchObservedRunningTime="2026-10-30 20:52:41.087287037 +0000 UTC m=+866.267183545" Oct 30 20:52:41.278994 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:41.278962 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28" Oct 30 20:52:45.095789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:45.091885 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" Oct 30 20:52:47.097671 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:47.097634 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" Oct 30 20:52:49.470395 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:49.470361 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2"] Oct 30 20:52:49.474561 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:49.474537 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2" Oct 30 20:52:49.484971 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:49.484944 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2"] Oct 30 20:52:49.485386 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:49.485344 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-95a684\"" Oct 30 20:52:49.485386 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:49.485362 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-95a684-dockercfg-749fk\"" Oct 30 20:52:49.499173 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:49.499139 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjc25\" (UniqueName: \"kubernetes.io/projected/bbbf9456-036c-4acb-9d73-662428928fa8-kube-api-access-mjc25\") pod \"payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2\" (UID: \"bbbf9456-036c-4acb-9d73-662428928fa8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2" Oct 30 20:52:49.499344 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:49.499205 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbf9456-036c-4acb-9d73-662428928fa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2\" (UID: \"bbbf9456-036c-4acb-9d73-662428928fa8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2" Oct 30 20:52:49.609791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:49.602565 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbf9456-036c-4acb-9d73-662428928fa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2\" (UID: \"bbbf9456-036c-4acb-9d73-662428928fa8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2" Oct 30 20:52:49.609791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:49.602875 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjc25\" (UniqueName: \"kubernetes.io/projected/bbbf9456-036c-4acb-9d73-662428928fa8-kube-api-access-mjc25\") pod \"payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2\" (UID: \"bbbf9456-036c-4acb-9d73-662428928fa8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2" Oct 30 20:52:49.628792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:49.624596 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbf9456-036c-4acb-9d73-662428928fa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2\" (UID: \"bbbf9456-036c-4acb-9d73-662428928fa8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2" Oct 30 20:52:49.662788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:49.656108 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjc25\" (UniqueName: \"kubernetes.io/projected/bbbf9456-036c-4acb-9d73-662428928fa8-kube-api-access-mjc25\") pod \"payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2\" (UID: \"bbbf9456-036c-4acb-9d73-662428928fa8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2" Oct 30 20:52:49.795102 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:49.795003 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2" Oct 30 20:52:50.140009 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:50.135390 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2"] Oct 30 20:52:50.198825 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:50.198752 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2" event={"ID":"bbbf9456-036c-4acb-9d73-662428928fa8","Type":"ContainerStarted","Data":"6027185e06c026366c4069cc268cd7150f8afc01cdcdc16aa55c9d9597a0ca36"} Oct 30 20:52:51.207125 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:51.207089 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2" event={"ID":"bbbf9456-036c-4acb-9d73-662428928fa8","Type":"ContainerStarted","Data":"f58f20e71e9cf1db366ef684c25757c88da0dac2948b39b3a25533200640921b"} Oct 30 20:52:51.207540 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:51.207190 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2" Oct 30 20:52:51.223536 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:51.223482 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2" podStartSLOduration=2.223466554 podStartE2EDuration="2.223466554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:51.223251036 +0000 UTC m=+876.403147537" watchObservedRunningTime="2026-10-30 20:52:51.223466554 +0000 UTC m=+876.403363062" Oct 30 20:52:52.514711 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:52:52.514662 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:52:52.672593 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.672547 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq"] Oct 30 20:52:52.678849 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.678826 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.686496 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.686293 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1982ff-openshift-default-dockercfg-n8lnr\"" Oct 30 20:52:52.692817 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.692787 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq"] Oct 30 20:52:52.699132 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.697128 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-credential-socket\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.699132 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.697243 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-workload-socket\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.699132 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.697351 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-workload-certs\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.699132 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.697386 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-token\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.699132 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.697433 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-envoy\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.699132 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.697487 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggmxj\" (UniqueName: \"kubernetes.io/projected/3199622d-dbd3-4e39-8376-93322d66b0ae-kube-api-access-ggmxj\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.699132 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.697641 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-data\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.699132 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.697697 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3199622d-dbd3-4e39-8376-93322d66b0ae-istiod-ca-cert\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.699132 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.697799 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-podinfo\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.805958 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.805873 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-workload-socket\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.811639 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.811606 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-workload-certs\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.812245 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.809373 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-workload-socket\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.812373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.812185 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-workload-certs\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.812373 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.812229 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-token\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.812969 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.812950 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-envoy\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.813086 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.813017 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggmxj\" (UniqueName: \"kubernetes.io/projected/3199622d-dbd3-4e39-8376-93322d66b0ae-kube-api-access-ggmxj\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.813086 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.813071 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-data\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.813200 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.813124 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3199622d-dbd3-4e39-8376-93322d66b0ae-istiod-ca-cert\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.813200 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.813165 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-podinfo\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.813354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.813337 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-credential-socket\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.813667 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.813650 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-credential-socket\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.817581 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.817546 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-data\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.868816 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.830242 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3199622d-dbd3-4e39-8376-93322d66b0ae-istiod-ca-cert\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.868816 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.866010 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-podinfo\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.896789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.870492 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggmxj\" (UniqueName: \"kubernetes.io/projected/3199622d-dbd3-4e39-8376-93322d66b0ae-kube-api-access-ggmxj\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.897629 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.897595 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-token\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:52.927794 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:52.926849 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-envoy\") pod \"e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:53.010786 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:53.006689 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:53.364269 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:53.364241 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq"] Oct 30 20:52:53.365057 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:52:53.365027 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3199622d_dbd3_4e39_8376_93322d66b0ae.slice/crio-f2abea9941b97d7daf92a991f0f425e48403ded93e40c8fdf108598c8550cab1 WatchSource:0}: Error finding container f2abea9941b97d7daf92a991f0f425e48403ded93e40c8fdf108598c8550cab1: Status 404 returned error can't find the container with id f2abea9941b97d7daf92a991f0f425e48403ded93e40c8fdf108598c8550cab1 Oct 30 20:52:53.367713 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:53.367682 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:53.367803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:53.367749 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:53.367803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:53.367798 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:52:54.238204 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:54.238162 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" event={"ID":"3199622d-dbd3-4e39-8376-93322d66b0ae","Type":"ContainerStarted","Data":"4c9a76b6be18f39f2948f7604574f1a3a5288c89ff0aa9225eb3cb2049a6fd34"} Oct 30 20:52:54.238832 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:54.238803 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" event={"ID":"3199622d-dbd3-4e39-8376-93322d66b0ae","Type":"ContainerStarted","Data":"f2abea9941b97d7daf92a991f0f425e48403ded93e40c8fdf108598c8550cab1"} Oct 30 20:52:54.272861 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:54.272795 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" podStartSLOduration=2.272761655 podStartE2EDuration="2.272761655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:52:54.264227354 +0000 UTC m=+879.444123862" watchObservedRunningTime="2026-10-30 20:52:54.272761655 +0000 UTC m=+879.452658163" Oct 30 20:52:55.023788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:55.016478 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:55.028477 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:55.028445 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:55.272926 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:55.272894 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:55.277406 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:55.277331 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:52:57.150903 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:57.145031 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv"] Oct 30 20:52:57.173136 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:57.173100 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" Oct 30 20:52:57.188824 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:57.188060 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-95a684-serving-cert\"" Oct 30 20:52:57.207023 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:57.206857 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv"] Oct 30 20:52:57.263049 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:57.263017 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2xgz\" (UniqueName: \"kubernetes.io/projected/338bef5a-21a8-4a89-8c84-539bdb24482b-kube-api-access-q2xgz\") pod \"maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv\" (UID: \"338bef5a-21a8-4a89-8c84-539bdb24482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" Oct 30 20:52:57.268901 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:57.268859 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/338bef5a-21a8-4a89-8c84-539bdb24482b-maas-api-tls\") pod \"maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv\" (UID: \"338bef5a-21a8-4a89-8c84-539bdb24482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" Oct 30 20:52:57.269203 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:57.269178 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/338bef5a-21a8-4a89-8c84-539bdb24482b-metrics-certs\") pod \"maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv\" (UID: \"338bef5a-21a8-4a89-8c84-539bdb24482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" Oct 30 20:52:57.276840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:57.269422 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/338bef5a-21a8-4a89-8c84-539bdb24482b-openshift-service-ca\") pod \"maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv\" (UID: \"338bef5a-21a8-4a89-8c84-539bdb24482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" Oct 30 20:52:57.407422 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:57.381850 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/338bef5a-21a8-4a89-8c84-539bdb24482b-openshift-service-ca\") pod \"maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv\" (UID: \"338bef5a-21a8-4a89-8c84-539bdb24482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" Oct 30 20:52:57.770511 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:57.739919 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2xgz\" (UniqueName: \"kubernetes.io/projected/338bef5a-21a8-4a89-8c84-539bdb24482b-kube-api-access-q2xgz\") pod \"maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv\" (UID: \"338bef5a-21a8-4a89-8c84-539bdb24482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" Oct 30 20:52:57.770511 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:57.739977 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/338bef5a-21a8-4a89-8c84-539bdb24482b-maas-api-tls\") pod \"maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv\" (UID: \"338bef5a-21a8-4a89-8c84-539bdb24482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" Oct 30 20:52:57.770511 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:57.740050 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/338bef5a-21a8-4a89-8c84-539bdb24482b-metrics-certs\") pod \"maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv\" (UID: \"338bef5a-21a8-4a89-8c84-539bdb24482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" Oct 30 20:52:57.924791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:57.418842 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/338bef5a-21a8-4a89-8c84-539bdb24482b-openshift-service-ca\") pod \"maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv\" (UID: \"338bef5a-21a8-4a89-8c84-539bdb24482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" Oct 30 20:52:57.924791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:57.737972 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:57.969793 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:52:57.969602 4848 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-serving-cert: secret "maas-api-e2e-api-a-95a684-serving-cert" not found Oct 30 20:52:58.144848 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:58.144733 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 20:52:58.169791 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:52:58.150380 4848 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/338bef5a-21a8-4a89-8c84-539bdb24482b-maas-api-tls podName:338bef5a-21a8-4a89-8c84-539bdb24482b nodeName:}" failed. No retries permitted until 2026-10-30 20:52:58.650352844 +0000 UTC m=+883.830249345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/338bef5a-21a8-4a89-8c84-539bdb24482b-maas-api-tls") pod "maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" (UID: "338bef5a-21a8-4a89-8c84-539bdb24482b") : secret "maas-api-e2e-api-a-95a684-serving-cert" not found Oct 30 20:52:58.194858 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:58.185042 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:58.208454 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:58.206893 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/338bef5a-21a8-4a89-8c84-539bdb24482b-metrics-certs\") pod \"maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv\" (UID: \"338bef5a-21a8-4a89-8c84-539bdb24482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" Oct 30 20:52:58.256838 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:58.243388 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:58.256838 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:58.245803 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2xgz\" (UniqueName: \"kubernetes.io/projected/338bef5a-21a8-4a89-8c84-539bdb24482b-kube-api-access-q2xgz\") pod \"maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv\" (UID: \"338bef5a-21a8-4a89-8c84-539bdb24482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" Oct 30 20:52:58.690678 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:58.690635 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/338bef5a-21a8-4a89-8c84-539bdb24482b-maas-api-tls\") pod \"maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv\" (UID: \"338bef5a-21a8-4a89-8c84-539bdb24482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" Oct 30 20:52:58.750793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:58.747796 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/338bef5a-21a8-4a89-8c84-539bdb24482b-maas-api-tls\") pod \"maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv\" (UID: \"338bef5a-21a8-4a89-8c84-539bdb24482b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" Oct 30 20:52:58.760797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:58.759864 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:58.963196 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:58.963102 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:58.974268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:58.974236 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:52:59.020263 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:59.020227 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" Oct 30 20:52:59.186102 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:59.186049 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:59.187362 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:59.187338 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:52:59.205606 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:59.204717 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:59.220404 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:59.206389 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:52:59.224647 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:59.224564 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:59.335165 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:59.324471 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 20:52:59.335165 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:59.324978 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:52:59.769201 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:59.769123 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:52:59.830793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:59.823547 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:52:59.934790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:59.933209 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6"] Oct 30 20:52:59.983942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:52:59.983899 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:53:00.177842 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:00.177262 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6"] Oct 30 20:53:00.177842 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:00.177398 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6" Oct 30 20:53:00.183377 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:00.183349 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1982ff-dockercfg-zf4k4\"" Oct 30 20:53:00.183631 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:00.183616 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1982ff\"" Oct 30 20:53:00.192633 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:00.187227 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:53:00.206420 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:00.206342 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:53:00.383321 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:00.316369 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c28ea7-d56c-47d6-9a10-531506720261-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6\" (UID: \"05c28ea7-d56c-47d6-9a10-531506720261\") " pod="openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6" Oct 30 20:53:00.383321 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:00.316575 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgvhz\" (UniqueName: \"kubernetes.io/projected/05c28ea7-d56c-47d6-9a10-531506720261-kube-api-access-pgvhz\") pod \"payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6\" (UID: \"05c28ea7-d56c-47d6-9a10-531506720261\") " pod="openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6" Oct 30 20:53:00.383321 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:00.325000 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:53:00.508846 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:00.420712 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgvhz\" (UniqueName: \"kubernetes.io/projected/05c28ea7-d56c-47d6-9a10-531506720261-kube-api-access-pgvhz\") pod \"payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6\" (UID: \"05c28ea7-d56c-47d6-9a10-531506720261\") " pod="openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6" Oct 30 20:53:00.508846 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:00.420931 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c28ea7-d56c-47d6-9a10-531506720261-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6\" (UID: \"05c28ea7-d56c-47d6-9a10-531506720261\") " pod="openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6" Oct 30 20:53:00.508846 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:00.422040 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c28ea7-d56c-47d6-9a10-531506720261-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6\" (UID: \"05c28ea7-d56c-47d6-9a10-531506720261\") " pod="openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6" Oct 30 20:53:00.651833 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:00.650205 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgvhz\" (UniqueName: \"kubernetes.io/projected/05c28ea7-d56c-47d6-9a10-531506720261-kube-api-access-pgvhz\") pod \"payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6\" (UID: \"05c28ea7-d56c-47d6-9a10-531506720261\") " pod="openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6" Oct 30 20:53:00.906330 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:00.773862 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:53:01.006791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:00.986021 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:53:01.047497 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:01.010725 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6" Oct 30 20:53:01.213091 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:01.213034 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:53:01.251585 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:01.251488 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:53:01.341913 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:01.341862 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 20:53:01.494914 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:01.494884 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv"] Oct 30 20:53:01.495184 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:01.495171 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 20:53:01.660030 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:01.659998 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:53:01.695361 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:01.695333 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 20:53:01.695977 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:01.695958 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:53:01.696230 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:01.696214 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:53:02.041169 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:02.041132 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6"] Oct 30 20:53:02.101668 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:02.101642 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6" event={"ID":"05c28ea7-d56c-47d6-9a10-531506720261","Type":"ContainerStarted","Data":"897fe383c5a98658e98a6298789c3ec43c5b342000339b47baf9753c2b2991c6"} Oct 30 20:53:02.115798 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:02.108654 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" event={"ID":"338bef5a-21a8-4a89-8c84-539bdb24482b","Type":"ContainerStarted","Data":"6edc7548dbb0570bb51e1413ad5c00c82bb372e44c4c456820161b94142cf956"} Oct 30 20:53:02.136076 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:02.135956 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" Oct 30 20:53:02.136076 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:02.136011 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" event={"ID":"338bef5a-21a8-4a89-8c84-539bdb24482b","Type":"ContainerStarted","Data":"251fc6d33d791dcb332f13b3049424f4cea36f2f30136c31aa6a248da7eeee84"} Oct 30 20:53:02.171222 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:02.153345 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" podStartSLOduration=4.429485001 podStartE2EDuration="5.153325695s" totalImagesPullingTime="723.840694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:02.148085224 +0000 UTC m=+887.327981749" watchObservedRunningTime="2026-10-30 20:53:02.153325695 +0000 UTC m=+887.333222204" Oct 30 20:53:02.201473 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:02.201441 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z"] Oct 30 20:53:02.236819 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:02.234284 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z" Oct 30 20:53:02.244366 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:02.242084 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-49990a\"" Oct 30 20:53:02.244366 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:02.242275 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-49990a-dockercfg-nnvsm\"" Oct 30 20:53:02.244366 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:02.242532 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2" Oct 30 20:53:02.245819 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:02.245793 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z"] Oct 30 20:53:02.291573 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:02.291534 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcxhq\" (UniqueName: \"kubernetes.io/projected/f616d8f8-b034-4199-8bb0-fe3999e85785-kube-api-access-bcxhq\") pod \"payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z\" (UID: \"f616d8f8-b034-4199-8bb0-fe3999e85785\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z" Oct 30 20:53:02.291950 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:02.291920 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f616d8f8-b034-4199-8bb0-fe3999e85785-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z\" (UID: \"f616d8f8-b034-4199-8bb0-fe3999e85785\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z" Oct 30 20:53:02.393613 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:02.393533 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f616d8f8-b034-4199-8bb0-fe3999e85785-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z\" (UID: \"f616d8f8-b034-4199-8bb0-fe3999e85785\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z" Oct 30 20:53:02.393785 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:02.393668 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcxhq\" (UniqueName: \"kubernetes.io/projected/f616d8f8-b034-4199-8bb0-fe3999e85785-kube-api-access-bcxhq\") pod \"payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z\" (UID: \"f616d8f8-b034-4199-8bb0-fe3999e85785\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z" Oct 30 20:53:02.394321 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:02.394300 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f616d8f8-b034-4199-8bb0-fe3999e85785-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z\" (UID: \"f616d8f8-b034-4199-8bb0-fe3999e85785\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z" Oct 30 20:53:02.402947 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:02.402917 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcxhq\" (UniqueName: \"kubernetes.io/projected/f616d8f8-b034-4199-8bb0-fe3999e85785-kube-api-access-bcxhq\") pod \"payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z\" (UID: \"f616d8f8-b034-4199-8bb0-fe3999e85785\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z" Oct 30 20:53:02.563314 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:02.563022 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z" Oct 30 20:53:03.034726 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:03.034632 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z"] Oct 30 20:53:03.039377 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:53:03.039344 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf616d8f8_b034_4199_8bb0_fe3999e85785.slice/crio-4f40b13efbc6e3cd48a45d8d66247d07c3464a5d420b07ba5065b313ef5657e5 WatchSource:0}: Error finding container 4f40b13efbc6e3cd48a45d8d66247d07c3464a5d420b07ba5065b313ef5657e5: Status 404 returned error can't find the container with id 4f40b13efbc6e3cd48a45d8d66247d07c3464a5d420b07ba5065b313ef5657e5 Oct 30 20:53:03.148871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:03.148841 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z" event={"ID":"f616d8f8-b034-4199-8bb0-fe3999e85785","Type":"ContainerStarted","Data":"4f40b13efbc6e3cd48a45d8d66247d07c3464a5d420b07ba5065b313ef5657e5"} Oct 30 20:53:03.152133 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:03.152106 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6" event={"ID":"05c28ea7-d56c-47d6-9a10-531506720261","Type":"ContainerStarted","Data":"f7e101f3445b39a2e25006f17fa5749d8f15da09ef5a50f6cc965f28da85cb3b"} Oct 30 20:53:03.152254 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:03.152184 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6" Oct 30 20:53:03.173038 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:03.172986 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6" podStartSLOduration=4.172968626 podStartE2EDuration="4.172968626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:03.168274164 +0000 UTC m=+888.348170671" watchObservedRunningTime="2026-10-30 20:53:03.172968626 +0000 UTC m=+888.352865134" Oct 30 20:53:04.163026 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:04.162976 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z" event={"ID":"f616d8f8-b034-4199-8bb0-fe3999e85785","Type":"ContainerStarted","Data":"bcc74a62eb2f96454a9a99b25b2b7f3771ad0d671cfe5e4a3a152d743b0b9ffa"} Oct 30 20:53:04.163491 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:04.163128 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z" Oct 30 20:53:04.182331 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:04.182272 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z" podStartSLOduration=2.182255268 podStartE2EDuration="2.182255268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:04.18117276 +0000 UTC m=+889.361069267" watchObservedRunningTime="2026-10-30 20:53:04.182255268 +0000 UTC m=+889.362151776" Oct 30 20:53:08.156939 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:08.156908 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" Oct 30 20:53:09.300789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.298142 4848 generic.go:356] "Generic (PLEG): container finished" podID="a79a6860-1766-4d6b-9bb7-ccfa69a02989" containerID="12f5e8040c139384dd8b5f1fea17ac812928b78f1d108401eca7208ac2caba64" exitCode=0 Oct 30 20:53:09.300789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.298270 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" event={"ID":"a79a6860-1766-4d6b-9bb7-ccfa69a02989","Type":"ContainerDied","Data":"12f5e8040c139384dd8b5f1fea17ac812928b78f1d108401eca7208ac2caba64"} Oct 30 20:53:09.300789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.298812 4848 scope.go:122] "RemoveContainer" containerID="12f5e8040c139384dd8b5f1fea17ac812928b78f1d108401eca7208ac2caba64" Oct 30 20:53:09.301828 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.301060 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:09.301828 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.301145 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:09.301828 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.301190 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:09.589277 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.589203 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c"] Oct 30 20:53:09.593511 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.593490 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" Oct 30 20:53:09.599444 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.599422 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-49990a-serving-cert\"" Oct 30 20:53:09.604351 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.604327 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c"] Oct 30 20:53:09.683751 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.683715 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnffp\" (UniqueName: \"kubernetes.io/projected/6c39353d-db8a-4650-bf95-3ca9bc3be519-kube-api-access-lnffp\") pod \"maas-api-e2e-api-b-49990a-57b6b89458-lh42c\" (UID: \"6c39353d-db8a-4650-bf95-3ca9bc3be519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" Oct 30 20:53:09.683942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.683832 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c39353d-db8a-4650-bf95-3ca9bc3be519-maas-api-tls\") pod \"maas-api-e2e-api-b-49990a-57b6b89458-lh42c\" (UID: \"6c39353d-db8a-4650-bf95-3ca9bc3be519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" Oct 30 20:53:09.683942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.683898 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c39353d-db8a-4650-bf95-3ca9bc3be519-openshift-service-ca\") pod \"maas-api-e2e-api-b-49990a-57b6b89458-lh42c\" (UID: \"6c39353d-db8a-4650-bf95-3ca9bc3be519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" Oct 30 20:53:09.684099 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.684082 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c39353d-db8a-4650-bf95-3ca9bc3be519-metrics-certs\") pod \"maas-api-e2e-api-b-49990a-57b6b89458-lh42c\" (UID: \"6c39353d-db8a-4650-bf95-3ca9bc3be519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" Oct 30 20:53:09.789796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.788629 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c39353d-db8a-4650-bf95-3ca9bc3be519-openshift-service-ca\") pod \"maas-api-e2e-api-b-49990a-57b6b89458-lh42c\" (UID: \"6c39353d-db8a-4650-bf95-3ca9bc3be519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" Oct 30 20:53:09.789796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.788831 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c39353d-db8a-4650-bf95-3ca9bc3be519-metrics-certs\") pod \"maas-api-e2e-api-b-49990a-57b6b89458-lh42c\" (UID: \"6c39353d-db8a-4650-bf95-3ca9bc3be519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" Oct 30 20:53:09.789796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.788971 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnffp\" (UniqueName: \"kubernetes.io/projected/6c39353d-db8a-4650-bf95-3ca9bc3be519-kube-api-access-lnffp\") pod \"maas-api-e2e-api-b-49990a-57b6b89458-lh42c\" (UID: \"6c39353d-db8a-4650-bf95-3ca9bc3be519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" Oct 30 20:53:09.789796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.789032 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c39353d-db8a-4650-bf95-3ca9bc3be519-maas-api-tls\") pod \"maas-api-e2e-api-b-49990a-57b6b89458-lh42c\" (UID: \"6c39353d-db8a-4650-bf95-3ca9bc3be519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" Oct 30 20:53:09.807786 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.790975 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c39353d-db8a-4650-bf95-3ca9bc3be519-openshift-service-ca\") pod \"maas-api-e2e-api-b-49990a-57b6b89458-lh42c\" (UID: \"6c39353d-db8a-4650-bf95-3ca9bc3be519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" Oct 30 20:53:09.807786 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.804356 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c39353d-db8a-4650-bf95-3ca9bc3be519-maas-api-tls\") pod \"maas-api-e2e-api-b-49990a-57b6b89458-lh42c\" (UID: \"6c39353d-db8a-4650-bf95-3ca9bc3be519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" Oct 30 20:53:09.807786 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.805544 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c39353d-db8a-4650-bf95-3ca9bc3be519-metrics-certs\") pod \"maas-api-e2e-api-b-49990a-57b6b89458-lh42c\" (UID: \"6c39353d-db8a-4650-bf95-3ca9bc3be519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" Oct 30 20:53:09.807786 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.806870 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnffp\" (UniqueName: \"kubernetes.io/projected/6c39353d-db8a-4650-bf95-3ca9bc3be519-kube-api-access-lnffp\") pod \"maas-api-e2e-api-b-49990a-57b6b89458-lh42c\" (UID: \"6c39353d-db8a-4650-bf95-3ca9bc3be519\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" Oct 30 20:53:09.909753 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:09.909666 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" Oct 30 20:53:10.195440 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:10.195412 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c"] Oct 30 20:53:10.196747 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:53:10.196720 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c39353d_db8a_4650_bf95_3ca9bc3be519.slice/crio-7ae44e899fcb57a9a5d9769b8bbce69961fa40eb13ddbf7f6062027319cab190 WatchSource:0}: Error finding container 7ae44e899fcb57a9a5d9769b8bbce69961fa40eb13ddbf7f6062027319cab190: Status 404 returned error can't find the container with id 7ae44e899fcb57a9a5d9769b8bbce69961fa40eb13ddbf7f6062027319cab190 Oct 30 20:53:10.370790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:10.367614 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" event={"ID":"6c39353d-db8a-4650-bf95-3ca9bc3be519","Type":"ContainerStarted","Data":"7ae44e899fcb57a9a5d9769b8bbce69961fa40eb13ddbf7f6062027319cab190"} Oct 30 20:53:10.389814 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:10.382306 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" event={"ID":"a79a6860-1766-4d6b-9bb7-ccfa69a02989","Type":"ContainerStarted","Data":"accc45a41f4e009c3fd07a4d60498e68b1ba5081e65f4d4673fb823d010e4c37"} Oct 30 20:53:11.009791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:11.005882 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:53:11.228951 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:11.228920 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:53:11.230369 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:11.230339 4848 patch_prober.go:29] interesting pod/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 20:53:11.230492 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:11.230396 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" podUID="a79a6860-1766-4d6b-9bb7-ccfa69a02989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 20:53:11.389845 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:11.389742 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" event={"ID":"6c39353d-db8a-4650-bf95-3ca9bc3be519","Type":"ContainerStarted","Data":"5a3631b5718a5f476496d3825d94829a9d65db2b3dcd28dd5e69cf362f4aaa72"} Oct 30 20:53:11.389845 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:11.389837 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" Oct 30 20:53:11.428821 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:11.426289 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" podStartSLOduration=1.966214814 podStartE2EDuration="2.426270155s" totalImagesPullingTime="460.055341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:11.425949265 +0000 UTC m=+896.605845774" watchObservedRunningTime="2026-10-30 20:53:11.426270155 +0000 UTC m=+896.606166672" Oct 30 20:53:12.226790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:12.226684 4848 patch_prober.go:29] interesting pod/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 20:53:12.226790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:12.226743 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" podUID="a79a6860-1766-4d6b-9bb7-ccfa69a02989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 20:53:13.221216 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.221175 4848 patch_prober.go:29] interesting pod/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 20:53:13.221832 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.221800 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" podUID="a79a6860-1766-4d6b-9bb7-ccfa69a02989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 20:53:13.286790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.281831 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9"] Oct 30 20:53:13.300007 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.299972 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9"] Oct 30 20:53:13.301032 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.300524 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.349793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.348513 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.349793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.348599 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.349793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.348674 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.349793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.348710 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8xh8\" (UniqueName: \"kubernetes.io/projected/2bdeff6d-2976-4534-8857-b04b966de5c4-kube-api-access-q8xh8\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.363801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.361950 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8"] Oct 30 20:53:13.366357 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.348759 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bdeff6d-2976-4534-8857-b04b966de5c4-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.366514 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.366408 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.366514 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.366450 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.366514 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.366506 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.366752 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.366551 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.368873 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.368356 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8"] Oct 30 20:53:13.368873 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.368437 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.473416 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.468262 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.473826 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.473778 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.474004 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.473988 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.474116 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.474101 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.474223 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.474210 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.474376 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.474346 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8xh8\" (UniqueName: \"kubernetes.io/projected/2bdeff6d-2976-4534-8857-b04b966de5c4-kube-api-access-q8xh8\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.474501 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.474486 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.474618 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.474605 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rml9s\" (UniqueName: \"kubernetes.io/projected/0dafd76f-6a6c-479f-86e3-b67f973cab0e-kube-api-access-rml9s\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.474780 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.474742 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bdeff6d-2976-4534-8857-b04b966de5c4-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.474907 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.474893 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.475236 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.475213 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.475419 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.475394 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.475583 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.475569 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.475753 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.475719 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.490434 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.490396 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.491059 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.491034 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.491282 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.491262 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.491411 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.491398 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.491710 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.488390 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.491856 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.476990 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.491952 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.487565 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bdeff6d-2976-4534-8857-b04b966de5c4-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.492062 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.487918 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.492174 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.488138 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.542789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.541827 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8xh8\" (UniqueName: \"kubernetes.io/projected/2bdeff6d-2976-4534-8857-b04b966de5c4-kube-api-access-q8xh8\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.542789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.542467 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.546395 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.546335 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.555789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.555029 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.593188 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.593148 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.593188 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.593191 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.593430 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.593222 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.593430 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.593255 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.593430 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.593282 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rml9s\" (UniqueName: \"kubernetes.io/projected/0dafd76f-6a6c-479f-86e3-b67f973cab0e-kube-api-access-rml9s\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.593430 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.593328 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.593430 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.593413 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.593747 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.593496 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.593747 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.593641 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.593747 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.593654 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.593747 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.593686 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.594053 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.593744 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.594053 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.593934 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.594219 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.594199 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.595647 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.595629 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.595983 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.595967 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.619790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.612308 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.638406 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.638348 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rml9s\" (UniqueName: \"kubernetes.io/projected/0dafd76f-6a6c-479f-86e3-b67f973cab0e-kube-api-access-rml9s\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:13.685972 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.685941 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:13.697718 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:13.697664 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:14.033061 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.031729 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9"] Oct 30 20:53:14.037140 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:53:14.037012 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bdeff6d_2976_4534_8857_b04b966de5c4.slice/crio-2375cc8fe7da1fc283ee859cd2ff086b10143f076306a4024453ac31b62be2f0 WatchSource:0}: Error finding container 2375cc8fe7da1fc283ee859cd2ff086b10143f076306a4024453ac31b62be2f0: Status 404 returned error can't find the container with id 2375cc8fe7da1fc283ee859cd2ff086b10143f076306a4024453ac31b62be2f0 Oct 30 20:53:14.037869 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.037844 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8"] Oct 30 20:53:14.038961 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:53:14.038858 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dafd76f_6a6c_479f_86e3_b67f973cab0e.slice/crio-5d422990a49a6aca2a1500f2ee52bb9cae04bace2b13c0b4751a8ccca27398da WatchSource:0}: Error finding container 5d422990a49a6aca2a1500f2ee52bb9cae04bace2b13c0b4751a8ccca27398da: Status 404 returned error can't find the container with id 5d422990a49a6aca2a1500f2ee52bb9cae04bace2b13c0b4751a8ccca27398da Oct 30 20:53:14.040020 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.039971 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:14.040117 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.040068 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:14.040117 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.040103 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:14.041903 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.041879 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:14.041983 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.041930 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:14.041983 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.041963 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:14.215396 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.215015 4848 patch_prober.go:29] interesting pod/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 20:53:14.215396 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.215066 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" podUID="a79a6860-1766-4d6b-9bb7-ccfa69a02989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 20:53:14.255789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.237999 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c"] Oct 30 20:53:14.287897 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.287821 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c"] Oct 30 20:53:14.288203 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.288145 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.299266 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.299061 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-84f901-openshift-default-dockercfg-9htn5\"" Oct 30 20:53:14.367611 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.367379 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-credential-socket\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.367611 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.367454 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvqzz\" (UniqueName: \"kubernetes.io/projected/feb15caf-387e-40ab-a8f8-d6292cf0b28b-kube-api-access-nvqzz\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.367611 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.367518 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-workload-certs\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.367611 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.367581 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-envoy\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.367962 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.367715 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-workload-socket\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.367962 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.367790 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-podinfo\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.367962 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.367866 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istiod-ca-cert\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.367962 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.367953 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-data\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.368161 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.368019 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-token\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.459406 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.459367 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" event={"ID":"0dafd76f-6a6c-479f-86e3-b67f973cab0e","Type":"ContainerStarted","Data":"7c370286b08a8341526e2c1122ab46701e35259978fd2b6f7207cf8cba5cc34b"} Oct 30 20:53:14.474006 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.473975 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" event={"ID":"0dafd76f-6a6c-479f-86e3-b67f973cab0e","Type":"ContainerStarted","Data":"5d422990a49a6aca2a1500f2ee52bb9cae04bace2b13c0b4751a8ccca27398da"} Oct 30 20:53:14.474200 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.473053 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-data\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.474287 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.472635 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-data\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.474425 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.474411 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-token\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.474612 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.474598 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-credential-socket\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.474736 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.474724 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvqzz\" (UniqueName: \"kubernetes.io/projected/feb15caf-387e-40ab-a8f8-d6292cf0b28b-kube-api-access-nvqzz\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.474887 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.474870 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-workload-certs\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.475051 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.475037 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-envoy\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.475167 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.475154 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-workload-socket\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.475301 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.475287 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-podinfo\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.475449 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.475436 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istiod-ca-cert\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.476271 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.476248 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istiod-ca-cert\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.477507 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.477485 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-credential-socket\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.496081 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.478010 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-workload-certs\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.496251 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.495320 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-workload-socket\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.545653 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.541695 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-envoy\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.545653 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.542349 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvqzz\" (UniqueName: \"kubernetes.io/projected/feb15caf-387e-40ab-a8f8-d6292cf0b28b-kube-api-access-nvqzz\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.545653 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.544129 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-token\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.603611 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.602893 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" podStartSLOduration=1.6028717399999999 podStartE2EDuration="1.60287174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:14.552309531 +0000 UTC m=+899.732206040" watchObservedRunningTime="2026-10-30 20:53:14.60287174 +0000 UTC m=+899.782768251" Oct 30 20:53:14.633866 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.610469 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" event={"ID":"2bdeff6d-2976-4534-8857-b04b966de5c4","Type":"ContainerStarted","Data":"93320dffc87b4a3d3e0c23dfdc27b402462a45a1eee0a23a9aa450c8e4b64243"} Oct 30 20:53:14.633866 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.610536 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" event={"ID":"2bdeff6d-2976-4534-8857-b04b966de5c4","Type":"ContainerStarted","Data":"2375cc8fe7da1fc283ee859cd2ff086b10143f076306a4024453ac31b62be2f0"} Oct 30 20:53:14.633866 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.622779 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-podinfo\") pod \"e2e-api-b-84f901-openshift-default-58866446d7-xfr7c\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.673399 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.673369 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:14.687531 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.686053 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:14.688107 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.688059 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" podStartSLOduration=1.688040792 podStartE2EDuration="1.688040792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:14.669447231 +0000 UTC m=+899.849343739" watchObservedRunningTime="2026-10-30 20:53:14.688040792 +0000 UTC m=+899.867937301" Oct 30 20:53:14.722911 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:14.722886 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:15.175119 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:15.174946 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6" Oct 30 20:53:15.177129 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:53:15.177105 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeb15caf_387e_40ab_a8f8_d6292cf0b28b.slice/crio-b40a8371b5d26b683e09d4ff9daaa4eece225d77bf3d2522f47887fcd09a3385 WatchSource:0}: Error finding container b40a8371b5d26b683e09d4ff9daaa4eece225d77bf3d2522f47887fcd09a3385: Status 404 returned error can't find the container with id b40a8371b5d26b683e09d4ff9daaa4eece225d77bf3d2522f47887fcd09a3385 Oct 30 20:53:15.182934 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:15.182894 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c"] Oct 30 20:53:15.183792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:15.183738 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:15.183967 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:15.183943 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:15.184043 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:15.183998 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:15.185406 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:15.185381 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z" Oct 30 20:53:15.240328 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:15.240284 4848 patch_prober.go:29] interesting pod/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 20:53:15.240495 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:15.240355 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" podUID="a79a6860-1766-4d6b-9bb7-ccfa69a02989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 20:53:15.273959 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:15.273856 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 20:53:15.277125 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:15.277081 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 20:53:15.633923 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:15.633888 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" event={"ID":"feb15caf-387e-40ab-a8f8-d6292cf0b28b","Type":"ContainerStarted","Data":"93404305a665ba336762add05864fe32554a5c83036b3c9ba6a1e3c6121b29e4"} Oct 30 20:53:15.635244 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:15.635209 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" event={"ID":"feb15caf-387e-40ab-a8f8-d6292cf0b28b","Type":"ContainerStarted","Data":"b40a8371b5d26b683e09d4ff9daaa4eece225d77bf3d2522f47887fcd09a3385"} Oct 30 20:53:15.661532 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:15.661470 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" podStartSLOduration=1.6614508749999999 podStartE2EDuration="1.661450875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:15.658166282 +0000 UTC m=+900.838062791" watchObservedRunningTime="2026-10-30 20:53:15.661450875 +0000 UTC m=+900.841347383" Oct 30 20:53:15.674608 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:15.674320 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:15.691931 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:15.691885 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:53:15.692128 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:15.691961 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" podUID="2bdeff6d-2976-4534-8857-b04b966de5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 20:53:15.714002 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:15.713859 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 20:53:15.714002 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:15.713929 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" podUID="0dafd76f-6a6c-479f-86e3-b67f973cab0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 20:53:16.497244 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:16.497194 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:53:16.500089 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:16.500067 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:53:16.700790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:16.690664 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:53:16.700790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:16.690730 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" podUID="2bdeff6d-2976-4534-8857-b04b966de5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 20:53:16.777853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:16.708823 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:16.777853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:16.709867 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 20:53:16.777853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:16.709918 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" podUID="0dafd76f-6a6c-479f-86e3-b67f973cab0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 20:53:16.943323 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:16.886567 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:16.943323 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:16.922538 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:17.402153 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:17.402120 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" Oct 30 20:53:17.687785 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:17.687651 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:53:17.687785 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:17.687702 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" podUID="2bdeff6d-2976-4534-8857-b04b966de5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 20:53:17.699927 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:17.699900 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 20:53:17.700071 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:17.699950 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" podUID="0dafd76f-6a6c-479f-86e3-b67f973cab0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 20:53:18.686525 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:18.686489 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:53:18.686814 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:18.686548 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" podUID="2bdeff6d-2976-4534-8857-b04b966de5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 20:53:18.698037 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:18.698009 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 20:53:18.698359 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:18.698058 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" podUID="0dafd76f-6a6c-479f-86e3-b67f973cab0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 20:53:19.689489 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:19.687205 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:53:19.689489 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:19.687275 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" podUID="2bdeff6d-2976-4534-8857-b04b966de5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 20:53:19.710814 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:19.706935 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 20:53:19.710814 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:19.707001 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" podUID="0dafd76f-6a6c-479f-86e3-b67f973cab0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 20:53:20.693635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:20.693192 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:53:20.693635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:20.693255 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" podUID="2bdeff6d-2976-4534-8857-b04b966de5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 20:53:20.707056 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:20.706934 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 20:53:20.707056 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:20.706994 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" podUID="0dafd76f-6a6c-479f-86e3-b67f973cab0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 20:53:21.689853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:21.688975 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:53:21.689853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:21.689039 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" podUID="2bdeff6d-2976-4534-8857-b04b966de5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 20:53:21.716792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:21.716175 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 20:53:21.716792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:21.716237 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" podUID="0dafd76f-6a6c-479f-86e3-b67f973cab0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 20:53:22.558789 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:53:22.557791 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:53:22.686787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:22.686309 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:53:22.686787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:22.686362 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" podUID="2bdeff6d-2976-4534-8857-b04b966de5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 20:53:22.698823 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:22.697963 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 20:53:22.698823 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:22.698010 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" podUID="0dafd76f-6a6c-479f-86e3-b67f973cab0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 20:53:23.688789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:23.688134 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 20:53:23.688789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:23.688196 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" podUID="2bdeff6d-2976-4534-8857-b04b966de5c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 20:53:24.666787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:24.666293 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:24.667591 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:24.667560 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:24.668795 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:24.668749 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:53:24.890801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:24.888048 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:24.890801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:24.890323 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:24.896844 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:24.896810 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:53:28.038200 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.038162 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5"] Oct 30 20:53:28.053559 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.053533 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5"] Oct 30 20:53:28.053916 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.053900 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.090369 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.090340 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0333a9-openshift-default-dockercfg-jfh9b\"" Oct 30 20:53:28.154297 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.154268 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-workload-certs\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.154297 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.154303 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-workload-socket\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.154518 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.154349 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-token\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.154565 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.154523 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-credential-socket\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.154605 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.154593 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-envoy\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.154698 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.154674 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-podinfo\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.154739 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.154726 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istiod-ca-cert\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.154805 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.154775 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-data\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.154849 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.154829 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msxz7\" (UniqueName: \"kubernetes.io/projected/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-kube-api-access-msxz7\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.262207 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.262166 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-credential-socket\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.262499 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.262479 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-envoy\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.262632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.262616 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-podinfo\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.262750 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.262736 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istiod-ca-cert\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.262882 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.262865 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-data\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.263009 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.262993 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msxz7\" (UniqueName: \"kubernetes.io/projected/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-kube-api-access-msxz7\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.263185 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.263165 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-workload-certs\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.285174 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.285144 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-workload-socket\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.285488 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.285454 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-token\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.286436 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.284039 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-workload-certs\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.323839 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.284646 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istiod-ca-cert\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.323839 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.285861 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-workload-socket\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.323839 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.264006 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-credential-socket\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.323839 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.264618 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-data\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.323839 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.288999 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-envoy\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.323839 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.302479 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-podinfo\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.330883 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.330820 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-token\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.331608 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.331581 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msxz7\" (UniqueName: \"kubernetes.io/projected/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-kube-api-access-msxz7\") pod \"e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.380325 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.380048 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:28.910130 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.910101 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5"] Oct 30 20:53:28.911266 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:53:28.911240 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd983d7e5_28e1_44a2_82ba_f2cd4f8ca5e4.slice/crio-8758eb869b723144beab15c4b2c05a7e3b636c47dfb282e1558d02d7b58e9c86 WatchSource:0}: Error finding container 8758eb869b723144beab15c4b2c05a7e3b636c47dfb282e1558d02d7b58e9c86: Status 404 returned error can't find the container with id 8758eb869b723144beab15c4b2c05a7e3b636c47dfb282e1558d02d7b58e9c86 Oct 30 20:53:28.914391 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.914351 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:28.914478 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.914437 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:28.914526 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:28.914482 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:29.201707 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:29.201614 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" event={"ID":"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4","Type":"ContainerStarted","Data":"250b55bf2b87b5be4f405ed58af04e3ecc551408a3c037b91fd77c4b3d92bd2e"} Oct 30 20:53:29.201707 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:29.201680 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" event={"ID":"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4","Type":"ContainerStarted","Data":"8758eb869b723144beab15c4b2c05a7e3b636c47dfb282e1558d02d7b58e9c86"} Oct 30 20:53:29.227405 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:29.227336 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" podStartSLOduration=1.227316597 podStartE2EDuration="1.227316597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:29.224050942 +0000 UTC m=+914.403947452" watchObservedRunningTime="2026-10-30 20:53:29.227316597 +0000 UTC m=+914.407213104" Oct 30 20:53:29.387206 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:29.386467 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:29.442789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:29.439122 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:30.213051 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:30.213018 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:30.214178 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:30.214148 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:53:33.193788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.190477 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk"] Oct 30 20:53:33.195460 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.195431 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk"] Oct 30 20:53:33.195605 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.195510 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" Oct 30 20:53:33.205887 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.205501 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-84f901-serving-cert\"" Oct 30 20:53:33.238627 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.238589 4848 generic.go:356] "Generic (PLEG): container finished" podID="3199622d-dbd3-4e39-8376-93322d66b0ae" containerID="4c9a76b6be18f39f2948f7604574f1a3a5288c89ff0aa9225eb3cb2049a6fd34" exitCode=0 Oct 30 20:53:33.238870 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.238657 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" event={"ID":"3199622d-dbd3-4e39-8376-93322d66b0ae","Type":"ContainerDied","Data":"4c9a76b6be18f39f2948f7604574f1a3a5288c89ff0aa9225eb3cb2049a6fd34"} Oct 30 20:53:33.239230 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.239212 4848 scope.go:122] "RemoveContainer" containerID="4c9a76b6be18f39f2948f7604574f1a3a5288c89ff0aa9225eb3cb2049a6fd34" Oct 30 20:53:33.241289 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.241259 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:33.241409 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.241328 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:33.241409 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.241360 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:33.397806 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.394843 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfqwf\" (UniqueName: \"kubernetes.io/projected/864fb9de-cedc-4aab-a5ab-c8e038c976fc-kube-api-access-mfqwf\") pod \"maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk\" (UID: \"864fb9de-cedc-4aab-a5ab-c8e038c976fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" Oct 30 20:53:33.397806 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.394990 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864fb9de-cedc-4aab-a5ab-c8e038c976fc-maas-api-tls\") pod \"maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk\" (UID: \"864fb9de-cedc-4aab-a5ab-c8e038c976fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" Oct 30 20:53:33.397806 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.395160 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864fb9de-cedc-4aab-a5ab-c8e038c976fc-openshift-service-ca\") pod \"maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk\" (UID: \"864fb9de-cedc-4aab-a5ab-c8e038c976fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" Oct 30 20:53:33.397806 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.395185 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864fb9de-cedc-4aab-a5ab-c8e038c976fc-metrics-certs\") pod \"maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk\" (UID: \"864fb9de-cedc-4aab-a5ab-c8e038c976fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" Oct 30 20:53:33.496378 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.496280 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864fb9de-cedc-4aab-a5ab-c8e038c976fc-maas-api-tls\") pod \"maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk\" (UID: \"864fb9de-cedc-4aab-a5ab-c8e038c976fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" Oct 30 20:53:33.496559 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.496426 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864fb9de-cedc-4aab-a5ab-c8e038c976fc-openshift-service-ca\") pod \"maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk\" (UID: \"864fb9de-cedc-4aab-a5ab-c8e038c976fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" Oct 30 20:53:33.496559 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.496453 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864fb9de-cedc-4aab-a5ab-c8e038c976fc-metrics-certs\") pod \"maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk\" (UID: \"864fb9de-cedc-4aab-a5ab-c8e038c976fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" Oct 30 20:53:33.496705 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.496684 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfqwf\" (UniqueName: \"kubernetes.io/projected/864fb9de-cedc-4aab-a5ab-c8e038c976fc-kube-api-access-mfqwf\") pod \"maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk\" (UID: \"864fb9de-cedc-4aab-a5ab-c8e038c976fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" Oct 30 20:53:33.497270 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.497246 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864fb9de-cedc-4aab-a5ab-c8e038c976fc-openshift-service-ca\") pod \"maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk\" (UID: \"864fb9de-cedc-4aab-a5ab-c8e038c976fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" Oct 30 20:53:33.499257 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.499227 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864fb9de-cedc-4aab-a5ab-c8e038c976fc-metrics-certs\") pod \"maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk\" (UID: \"864fb9de-cedc-4aab-a5ab-c8e038c976fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" Oct 30 20:53:33.499447 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.499427 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864fb9de-cedc-4aab-a5ab-c8e038c976fc-maas-api-tls\") pod \"maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk\" (UID: \"864fb9de-cedc-4aab-a5ab-c8e038c976fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" Oct 30 20:53:33.505876 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.505849 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfqwf\" (UniqueName: \"kubernetes.io/projected/864fb9de-cedc-4aab-a5ab-c8e038c976fc-kube-api-access-mfqwf\") pod \"maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk\" (UID: \"864fb9de-cedc-4aab-a5ab-c8e038c976fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" Oct 30 20:53:33.513569 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.513549 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" Oct 30 20:53:33.848304 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:33.848269 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk"] Oct 30 20:53:33.849412 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:53:33.849382 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864fb9de_cedc_4aab_a5ab_c8e038c976fc.slice/crio-0d4683383a8a20d51bc2fe8cbffc36fa21c053b232ca4d64d8718847de2b9c40 WatchSource:0}: Error finding container 0d4683383a8a20d51bc2fe8cbffc36fa21c053b232ca4d64d8718847de2b9c40: Status 404 returned error can't find the container with id 0d4683383a8a20d51bc2fe8cbffc36fa21c053b232ca4d64d8718847de2b9c40 Oct 30 20:53:34.304790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:34.304733 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" event={"ID":"864fb9de-cedc-4aab-a5ab-c8e038c976fc","Type":"ContainerStarted","Data":"0d4683383a8a20d51bc2fe8cbffc36fa21c053b232ca4d64d8718847de2b9c40"} Oct 30 20:53:34.395381 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:34.395345 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" event={"ID":"3199622d-dbd3-4e39-8376-93322d66b0ae","Type":"ContainerStarted","Data":"b52c654dd6100dd368a83aef50156141cd132e54710968714399cc19cde86289"} Oct 30 20:53:35.027363 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:35.027325 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:53:35.029242 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:35.029207 4848 patch_prober.go:29] interesting pod/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 20:53:35.029397 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:35.029268 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" podUID="3199622d-dbd3-4e39-8376-93322d66b0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 20:53:35.483794 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:35.483161 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" event={"ID":"864fb9de-cedc-4aab-a5ab-c8e038c976fc","Type":"ContainerStarted","Data":"748bc469ff35c832ea691673603a755e50f31b4cfb5c0e85ceea7cf6048736c0"} Oct 30 20:53:35.483794 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:35.483247 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" Oct 30 20:53:36.028976 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:36.027386 4848 patch_prober.go:29] interesting pod/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 20:53:36.028976 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:36.027445 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" podUID="3199622d-dbd3-4e39-8376-93322d66b0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 20:53:37.028790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:37.028440 4848 patch_prober.go:29] interesting pod/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 20:53:37.028790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:37.028501 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" podUID="3199622d-dbd3-4e39-8376-93322d66b0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 20:53:38.033802 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:38.031448 4848 patch_prober.go:29] interesting pod/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 20:53:38.033802 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:38.031509 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" podUID="3199622d-dbd3-4e39-8376-93322d66b0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 20:53:39.014160 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:39.014125 4848 patch_prober.go:29] interesting pod/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 20:53:39.014442 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:39.014411 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" podUID="3199622d-dbd3-4e39-8376-93322d66b0ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 20:53:40.018055 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:40.018020 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:53:40.018529 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:40.018513 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:53:40.019225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:40.019204 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:53:40.062967 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:40.062908 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" podStartSLOduration=7.459502976 podStartE2EDuration="8.062888853s" totalImagesPullingTime="603.385877ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:35.520119167 +0000 UTC m=+920.700015674" watchObservedRunningTime="2026-10-30 20:53:40.062888853 +0000 UTC m=+925.242785366" Oct 30 20:53:40.303821 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:40.303717 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28"] Oct 30 20:53:42.563651 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:42.563615 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" Oct 30 20:53:42.805863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:42.805822 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc"] Oct 30 20:53:42.810467 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:42.810437 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" Oct 30 20:53:42.812400 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:42.812376 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd234ce-077b-40b6-ba44-aec166a415aa-maas-api-tls\") pod \"maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc\" (UID: \"3cd234ce-077b-40b6-ba44-aec166a415aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" Oct 30 20:53:42.812633 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:42.812611 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd234ce-077b-40b6-ba44-aec166a415aa-openshift-service-ca\") pod \"maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc\" (UID: \"3cd234ce-077b-40b6-ba44-aec166a415aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" Oct 30 20:53:42.812837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:42.812820 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7vhc\" (UniqueName: \"kubernetes.io/projected/3cd234ce-077b-40b6-ba44-aec166a415aa-kube-api-access-r7vhc\") pod \"maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc\" (UID: \"3cd234ce-077b-40b6-ba44-aec166a415aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" Oct 30 20:53:42.813066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:42.813047 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd234ce-077b-40b6-ba44-aec166a415aa-metrics-certs\") pod \"maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc\" (UID: \"3cd234ce-077b-40b6-ba44-aec166a415aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" Oct 30 20:53:42.829956 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:42.829887 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0333a9-serving-cert\"" Oct 30 20:53:42.845786 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:42.837379 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc"] Oct 30 20:53:42.927137 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:42.916804 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7vhc\" (UniqueName: \"kubernetes.io/projected/3cd234ce-077b-40b6-ba44-aec166a415aa-kube-api-access-r7vhc\") pod \"maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc\" (UID: \"3cd234ce-077b-40b6-ba44-aec166a415aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" Oct 30 20:53:42.927137 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:42.927044 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd234ce-077b-40b6-ba44-aec166a415aa-metrics-certs\") pod \"maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc\" (UID: \"3cd234ce-077b-40b6-ba44-aec166a415aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" Oct 30 20:53:42.927396 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:42.927368 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd234ce-077b-40b6-ba44-aec166a415aa-maas-api-tls\") pod \"maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc\" (UID: \"3cd234ce-077b-40b6-ba44-aec166a415aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" Oct 30 20:53:42.927454 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:42.927422 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd234ce-077b-40b6-ba44-aec166a415aa-openshift-service-ca\") pod \"maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc\" (UID: \"3cd234ce-077b-40b6-ba44-aec166a415aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" Oct 30 20:53:42.928815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:42.928516 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd234ce-077b-40b6-ba44-aec166a415aa-openshift-service-ca\") pod \"maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc\" (UID: \"3cd234ce-077b-40b6-ba44-aec166a415aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" Oct 30 20:53:42.944603 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:42.944566 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd234ce-077b-40b6-ba44-aec166a415aa-metrics-certs\") pod \"maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc\" (UID: \"3cd234ce-077b-40b6-ba44-aec166a415aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" Oct 30 20:53:42.949481 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:42.949445 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7vhc\" (UniqueName: \"kubernetes.io/projected/3cd234ce-077b-40b6-ba44-aec166a415aa-kube-api-access-r7vhc\") pod \"maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc\" (UID: \"3cd234ce-077b-40b6-ba44-aec166a415aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" Oct 30 20:53:42.950287 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:42.950260 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd234ce-077b-40b6-ba44-aec166a415aa-maas-api-tls\") pod \"maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc\" (UID: \"3cd234ce-077b-40b6-ba44-aec166a415aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" Oct 30 20:53:43.155841 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:43.151244 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" Oct 30 20:53:43.692817 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:43.685596 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc"] Oct 30 20:53:44.646440 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:44.646347 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" event={"ID":"3cd234ce-077b-40b6-ba44-aec166a415aa","Type":"ContainerStarted","Data":"8a88e6433b3b97d28ea4fb42b23951469aa9b9e64e59c9014882a4a64eb1835c"} Oct 30 20:53:44.646440 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:44.646405 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" event={"ID":"3cd234ce-077b-40b6-ba44-aec166a415aa","Type":"ContainerStarted","Data":"cd3543090a86c7bf34e23d356e1dd359126be835ab4efb83c148a62820bca225"} Oct 30 20:53:44.646639 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:44.646579 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" Oct 30 20:53:44.669642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:44.669593 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" podStartSLOduration=2.097052835 podStartE2EDuration="2.669578343s" totalImagesPullingTime="572.525508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:53:44.664547696 +0000 UTC m=+929.844444203" watchObservedRunningTime="2026-10-30 20:53:44.669578343 +0000 UTC m=+929.849474850" Oct 30 20:53:45.307058 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:45.304858 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28" podUID="2c6e17b1-e385-470d-89f0-25397789a650" containerName="payload-processing" containerID="cri-o://5cc24cc74784ac6d48b40388cb712779c76b86db237e92629044974f873ab3c5" gracePeriod=25 Oct 30 20:53:45.638878 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:45.638852 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28" Oct 30 20:53:45.657538 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:45.657496 4848 generic.go:356] "Generic (PLEG): container finished" podID="2c6e17b1-e385-470d-89f0-25397789a650" containerID="5cc24cc74784ac6d48b40388cb712779c76b86db237e92629044974f873ab3c5" exitCode=0 Oct 30 20:53:45.657708 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:45.657540 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28" event={"ID":"2c6e17b1-e385-470d-89f0-25397789a650","Type":"ContainerDied","Data":"5cc24cc74784ac6d48b40388cb712779c76b86db237e92629044974f873ab3c5"} Oct 30 20:53:45.657708 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:45.657575 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28" Oct 30 20:53:45.657708 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:45.657597 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28" event={"ID":"2c6e17b1-e385-470d-89f0-25397789a650","Type":"ContainerDied","Data":"ec6b4354df17a841225013da067600248ecf2e397920064d28e1455c2e18a3d1"} Oct 30 20:53:45.657708 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:45.657627 4848 scope.go:122] "RemoveContainer" containerID="5cc24cc74784ac6d48b40388cb712779c76b86db237e92629044974f873ab3c5" Oct 30 20:53:45.667564 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:45.667527 4848 scope.go:122] "RemoveContainer" containerID="5cc24cc74784ac6d48b40388cb712779c76b86db237e92629044974f873ab3c5" Oct 30 20:53:45.667888 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:53:45.667864 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc24cc74784ac6d48b40388cb712779c76b86db237e92629044974f873ab3c5\": container with ID starting with 5cc24cc74784ac6d48b40388cb712779c76b86db237e92629044974f873ab3c5 not found: ID does not exist" containerID="5cc24cc74784ac6d48b40388cb712779c76b86db237e92629044974f873ab3c5" Oct 30 20:53:45.667973 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:45.667895 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc24cc74784ac6d48b40388cb712779c76b86db237e92629044974f873ab3c5"} err="failed to get container status \"5cc24cc74784ac6d48b40388cb712779c76b86db237e92629044974f873ab3c5\": rpc error: code = NotFound desc = could not find container \"5cc24cc74784ac6d48b40388cb712779c76b86db237e92629044974f873ab3c5\": container with ID starting with 5cc24cc74784ac6d48b40388cb712779c76b86db237e92629044974f873ab3c5 not found: ID does not exist" Oct 30 20:53:45.762952 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:45.762916 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c6e17b1-e385-470d-89f0-25397789a650-kube-api-access-m7hjd\" (UniqueName: \"kubernetes.io/projected/2c6e17b1-e385-470d-89f0-25397789a650-kube-api-access-m7hjd\") pod \"2c6e17b1-e385-470d-89f0-25397789a650\" (UID: \"2c6e17b1-e385-470d-89f0-25397789a650\") " Oct 30 20:53:45.763121 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:45.763045 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c6e17b1-e385-470d-89f0-25397789a650-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6e17b1-e385-470d-89f0-25397789a650-plugins-config-volume\") pod \"2c6e17b1-e385-470d-89f0-25397789a650\" (UID: \"2c6e17b1-e385-470d-89f0-25397789a650\") " Oct 30 20:53:45.763430 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:45.763404 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c6e17b1-e385-470d-89f0-25397789a650-plugins-config-volume" pod "2c6e17b1-e385-470d-89f0-25397789a650" (UID: "2c6e17b1-e385-470d-89f0-25397789a650"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:53:45.763656 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:45.763634 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c6e17b1-e385-470d-89f0-25397789a650-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.765112 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:45.765091 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c6e17b1-e385-470d-89f0-25397789a650-kube-api-access-m7hjd" pod "2c6e17b1-e385-470d-89f0-25397789a650" (UID: "2c6e17b1-e385-470d-89f0-25397789a650"). InnerVolumeSpecName "kube-api-access-m7hjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:53:45.865071 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:45.864992 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7hjd\" (UniqueName: \"kubernetes.io/projected/2c6e17b1-e385-470d-89f0-25397789a650-kube-api-access-m7hjd\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:53:45.982286 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:45.982244 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28"] Oct 30 20:53:45.989505 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:45.989478 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fac16-cb48df6c-vtf28"] Oct 30 20:53:47.015792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:47.015020 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c6e17b1-e385-470d-89f0-25397789a650" path="/var/lib/kubelet/pods/2c6e17b1-e385-470d-89f0-25397789a650/volumes" Oct 30 20:53:50.664335 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:50.664300 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" Oct 30 20:53:52.518227 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:53:52.518154 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:53:53.722002 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:53.721968 4848 generic.go:356] "Generic (PLEG): container finished" podID="feb15caf-387e-40ab-a8f8-d6292cf0b28b" containerID="93404305a665ba336762add05864fe32554a5c83036b3c9ba6a1e3c6121b29e4" exitCode=0 Oct 30 20:53:53.722447 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:53.722037 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" event={"ID":"feb15caf-387e-40ab-a8f8-d6292cf0b28b","Type":"ContainerDied","Data":"93404305a665ba336762add05864fe32554a5c83036b3c9ba6a1e3c6121b29e4"} Oct 30 20:53:53.722447 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:53.722434 4848 scope.go:122] "RemoveContainer" containerID="93404305a665ba336762add05864fe32554a5c83036b3c9ba6a1e3c6121b29e4" Oct 30 20:53:53.724256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:53.724222 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:53.724376 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:53.724325 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:53.724436 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:53.724401 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:53:54.740674 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:54.740636 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" event={"ID":"feb15caf-387e-40ab-a8f8-d6292cf0b28b","Type":"ContainerStarted","Data":"3692cbefcc56c0cf2db24005b033c48a2c36896790c1a37375ec4e0d2770bae4"} Oct 30 20:53:55.674388 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:55.674354 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:53:55.674579 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:55.674475 4848 patch_prober.go:29] interesting pod/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 20:53:55.674579 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:55.674527 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" podUID="feb15caf-387e-40ab-a8f8-d6292cf0b28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 20:53:56.674524 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:56.674493 4848 patch_prober.go:29] interesting pod/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 20:53:56.674932 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:56.674551 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" podUID="feb15caf-387e-40ab-a8f8-d6292cf0b28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 20:53:57.674688 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:57.674645 4848 patch_prober.go:29] interesting pod/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 20:53:57.675197 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:57.674708 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" podUID="feb15caf-387e-40ab-a8f8-d6292cf0b28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 20:53:58.674775 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:58.674726 4848 patch_prober.go:29] interesting pod/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 20:53:58.675245 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:58.674812 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" podUID="feb15caf-387e-40ab-a8f8-d6292cf0b28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 20:53:59.633237 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:59.633194 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2"] Oct 30 20:53:59.674449 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:59.674416 4848 patch_prober.go:29] interesting pod/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 20:53:59.674721 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:53:59.674696 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" podUID="feb15caf-387e-40ab-a8f8-d6292cf0b28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 20:54:00.695816 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:00.695756 4848 patch_prober.go:29] interesting pod/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 20:54:00.711981 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:00.709016 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" podUID="feb15caf-387e-40ab-a8f8-d6292cf0b28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 20:54:01.708793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:01.708093 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:54:01.712793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:01.709585 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:54:01.718472 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:01.718447 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:54:02.320559 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:02.320520 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb"] Oct 30 20:54:02.320867 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:02.320836 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" podUID="706abb91-d2cd-4f79-8247-2f9077a602c6" containerName="maas-api" containerID="cri-o://58a60bdfad2954ce7a1763b7544f78ffae9cd88796159667a26f7cd0d8d6c3eb" gracePeriod=30 Oct 30 20:54:03.078550 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:03.078505 4848 generic.go:356] "Generic (PLEG): container finished" podID="706abb91-d2cd-4f79-8247-2f9077a602c6" containerID="58a60bdfad2954ce7a1763b7544f78ffae9cd88796159667a26f7cd0d8d6c3eb" exitCode=0 Oct 30 20:54:03.079996 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:03.079083 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" event={"ID":"706abb91-d2cd-4f79-8247-2f9077a602c6","Type":"ContainerDied","Data":"58a60bdfad2954ce7a1763b7544f78ffae9cd88796159667a26f7cd0d8d6c3eb"} Oct 30 20:54:03.080212 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:03.080190 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" event={"ID":"706abb91-d2cd-4f79-8247-2f9077a602c6","Type":"ContainerDied","Data":"7fd5f75fc441a972071eee1df42833aaf89cc0440db3f25206a6a9318d6e2f3a"} Oct 30 20:54:03.080316 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:03.080304 4848 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fd5f75fc441a972071eee1df42833aaf89cc0440db3f25206a6a9318d6e2f3a" Oct 30 20:54:03.082948 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:03.082931 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" Oct 30 20:54:03.138127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:03.138093 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/706abb91-d2cd-4f79-8247-2f9077a602c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/706abb91-d2cd-4f79-8247-2f9077a602c6-metrics-certs\") pod \"706abb91-d2cd-4f79-8247-2f9077a602c6\" (UID: \"706abb91-d2cd-4f79-8247-2f9077a602c6\") " Oct 30 20:54:03.138294 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:03.138265 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/706abb91-d2cd-4f79-8247-2f9077a602c6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/706abb91-d2cd-4f79-8247-2f9077a602c6-openshift-service-ca\") pod \"706abb91-d2cd-4f79-8247-2f9077a602c6\" (UID: \"706abb91-d2cd-4f79-8247-2f9077a602c6\") " Oct 30 20:54:03.138376 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:03.138327 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/706abb91-d2cd-4f79-8247-2f9077a602c6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706abb91-d2cd-4f79-8247-2f9077a602c6-maas-api-tls\") pod \"706abb91-d2cd-4f79-8247-2f9077a602c6\" (UID: \"706abb91-d2cd-4f79-8247-2f9077a602c6\") " Oct 30 20:54:03.138449 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:03.138398 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/706abb91-d2cd-4f79-8247-2f9077a602c6-kube-api-access-bmcm9\" (UniqueName: \"kubernetes.io/projected/706abb91-d2cd-4f79-8247-2f9077a602c6-kube-api-access-bmcm9\") pod \"706abb91-d2cd-4f79-8247-2f9077a602c6\" (UID: \"706abb91-d2cd-4f79-8247-2f9077a602c6\") " Oct 30 20:54:03.138700 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:03.138656 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/706abb91-d2cd-4f79-8247-2f9077a602c6-openshift-service-ca" pod "706abb91-d2cd-4f79-8247-2f9077a602c6" (UID: "706abb91-d2cd-4f79-8247-2f9077a602c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:03.139426 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:03.139144 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/706abb91-d2cd-4f79-8247-2f9077a602c6-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:03.140736 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:03.140710 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/706abb91-d2cd-4f79-8247-2f9077a602c6-metrics-certs" pod "706abb91-d2cd-4f79-8247-2f9077a602c6" (UID: "706abb91-d2cd-4f79-8247-2f9077a602c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:03.141319 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:03.141292 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/706abb91-d2cd-4f79-8247-2f9077a602c6-maas-api-tls" pod "706abb91-d2cd-4f79-8247-2f9077a602c6" (UID: "706abb91-d2cd-4f79-8247-2f9077a602c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:03.141403 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:03.141348 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706abb91-d2cd-4f79-8247-2f9077a602c6-kube-api-access-bmcm9" pod "706abb91-d2cd-4f79-8247-2f9077a602c6" (UID: "706abb91-d2cd-4f79-8247-2f9077a602c6"). InnerVolumeSpecName "kube-api-access-bmcm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:03.241415 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:03.241333 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/706abb91-d2cd-4f79-8247-2f9077a602c6-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:03.241415 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:03.241371 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmcm9\" (UniqueName: \"kubernetes.io/projected/706abb91-d2cd-4f79-8247-2f9077a602c6-kube-api-access-bmcm9\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:03.241415 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:03.241386 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/706abb91-d2cd-4f79-8247-2f9077a602c6-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:04.120288 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:04.120246 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb" Oct 30 20:54:04.181586 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:04.181552 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb"] Oct 30 20:54:04.188111 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:04.188082 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1fac16-f8d55cf99-g5mmb"] Oct 30 20:54:04.634742 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:04.634695 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2" podUID="bbbf9456-036c-4acb-9d73-662428928fa8" containerName="payload-pre-processing" containerID="cri-o://f58f20e71e9cf1db366ef684c25757c88da0dac2948b39b3a25533200640921b" gracePeriod=25 Oct 30 20:54:04.939582 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:04.939389 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="706abb91-d2cd-4f79-8247-2f9077a602c6" path="/var/lib/kubelet/pods/706abb91-d2cd-4f79-8247-2f9077a602c6/volumes" Oct 30 20:54:04.970449 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:04.970418 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2" Oct 30 20:54:05.082442 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:05.082399 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbf9456-036c-4acb-9d73-662428928fa8-kube-api-access-mjc25\" (UniqueName: \"kubernetes.io/projected/bbbf9456-036c-4acb-9d73-662428928fa8-kube-api-access-mjc25\") pod \"bbbf9456-036c-4acb-9d73-662428928fa8\" (UID: \"bbbf9456-036c-4acb-9d73-662428928fa8\") " Oct 30 20:54:05.082638 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:05.082595 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbf9456-036c-4acb-9d73-662428928fa8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbf9456-036c-4acb-9d73-662428928fa8-plugins-config-volume\") pod \"bbbf9456-036c-4acb-9d73-662428928fa8\" (UID: \"bbbf9456-036c-4acb-9d73-662428928fa8\") " Oct 30 20:54:05.083221 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:05.083189 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbf9456-036c-4acb-9d73-662428928fa8-plugins-config-volume" pod "bbbf9456-036c-4acb-9d73-662428928fa8" (UID: "bbbf9456-036c-4acb-9d73-662428928fa8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:05.083462 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:05.083440 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbbf9456-036c-4acb-9d73-662428928fa8-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:05.085527 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:05.085493 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbf9456-036c-4acb-9d73-662428928fa8-kube-api-access-mjc25" pod "bbbf9456-036c-4acb-9d73-662428928fa8" (UID: "bbbf9456-036c-4acb-9d73-662428928fa8"). InnerVolumeSpecName "kube-api-access-mjc25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:05.129435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:05.129395 4848 generic.go:356] "Generic (PLEG): container finished" podID="bbbf9456-036c-4acb-9d73-662428928fa8" containerID="f58f20e71e9cf1db366ef684c25757c88da0dac2948b39b3a25533200640921b" exitCode=0 Oct 30 20:54:05.129933 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:05.129442 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2" Oct 30 20:54:05.129933 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:05.129474 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2" event={"ID":"bbbf9456-036c-4acb-9d73-662428928fa8","Type":"ContainerDied","Data":"f58f20e71e9cf1db366ef684c25757c88da0dac2948b39b3a25533200640921b"} Oct 30 20:54:05.129933 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:05.129527 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2" event={"ID":"bbbf9456-036c-4acb-9d73-662428928fa8","Type":"ContainerDied","Data":"6027185e06c026366c4069cc268cd7150f8afc01cdcdc16aa55c9d9597a0ca36"} Oct 30 20:54:05.129933 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:05.129549 4848 scope.go:122] "RemoveContainer" containerID="f58f20e71e9cf1db366ef684c25757c88da0dac2948b39b3a25533200640921b" Oct 30 20:54:05.198776 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:05.187168 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjc25\" (UniqueName: \"kubernetes.io/projected/bbbf9456-036c-4acb-9d73-662428928fa8-kube-api-access-mjc25\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:05.235222 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:05.234848 4848 scope.go:122] "RemoveContainer" containerID="f58f20e71e9cf1db366ef684c25757c88da0dac2948b39b3a25533200640921b" Oct 30 20:54:05.235648 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:54:05.235608 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f58f20e71e9cf1db366ef684c25757c88da0dac2948b39b3a25533200640921b\": container with ID starting with f58f20e71e9cf1db366ef684c25757c88da0dac2948b39b3a25533200640921b not found: ID does not exist" containerID="f58f20e71e9cf1db366ef684c25757c88da0dac2948b39b3a25533200640921b" Oct 30 20:54:05.236180 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:05.236128 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f58f20e71e9cf1db366ef684c25757c88da0dac2948b39b3a25533200640921b"} err="failed to get container status \"f58f20e71e9cf1db366ef684c25757c88da0dac2948b39b3a25533200640921b\": rpc error: code = NotFound desc = could not find container \"f58f20e71e9cf1db366ef684c25757c88da0dac2948b39b3a25533200640921b\": container with ID starting with f58f20e71e9cf1db366ef684c25757c88da0dac2948b39b3a25533200640921b not found: ID does not exist" Oct 30 20:54:05.271886 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:05.271849 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2"] Oct 30 20:54:05.274116 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:05.274088 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-95a684-56fb6cb85f-sjhp2"] Oct 30 20:54:06.942235 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:06.942204 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbf9456-036c-4acb-9d73-662428928fa8" path="/var/lib/kubelet/pods/bbbf9456-036c-4acb-9d73-662428928fa8/volumes" Oct 30 20:54:07.147408 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:07.147326 4848 generic.go:356] "Generic (PLEG): container finished" podID="d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" containerID="250b55bf2b87b5be4f405ed58af04e3ecc551408a3c037b91fd77c4b3d92bd2e" exitCode=0 Oct 30 20:54:07.147408 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:07.147397 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" event={"ID":"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4","Type":"ContainerDied","Data":"250b55bf2b87b5be4f405ed58af04e3ecc551408a3c037b91fd77c4b3d92bd2e"} Oct 30 20:54:07.147960 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:07.147938 4848 scope.go:122] "RemoveContainer" containerID="250b55bf2b87b5be4f405ed58af04e3ecc551408a3c037b91fd77c4b3d92bd2e" Oct 30 20:54:07.149554 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:07.149530 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:54:07.149611 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:07.149595 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:54:07.149647 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:07.149625 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:54:07.838475 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:07.838433 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6"] Oct 30 20:54:08.217064 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:08.217026 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" event={"ID":"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4","Type":"ContainerStarted","Data":"06380184dc77630cbccc9555f3684de060f50eafa66b0efe759d2b7a95d5b991"} Oct 30 20:54:08.387245 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:08.387214 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:54:08.387611 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:08.387523 4848 patch_prober.go:29] interesting pod/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:54:08.387797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:08.387757 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" podUID="d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 20:54:09.381692 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:09.381653 4848 patch_prober.go:29] interesting pod/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:54:09.382198 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:09.381721 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" podUID="d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 20:54:10.151810 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:10.151740 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl"] Oct 30 20:54:10.153779 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:10.151990 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" podUID="a79a6860-1766-4d6b-9bb7-ccfa69a02989" containerName="istio-proxy" containerID="cri-o://accc45a41f4e009c3fd07a4d60498e68b1ba5081e65f4d4673fb823d010e4c37" gracePeriod=30 Oct 30 20:54:10.390385 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:10.390348 4848 patch_prober.go:29] interesting pod/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:54:10.391002 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:10.390967 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" podUID="d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 20:54:11.381313 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:11.381270 4848 patch_prober.go:29] interesting pod/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:54:11.381493 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:11.381335 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" podUID="d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 20:54:12.381676 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:12.381638 4848 patch_prober.go:29] interesting pod/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:54:12.382131 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:12.381722 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" podUID="d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 20:54:12.844838 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:12.841820 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6" podUID="05c28ea7-d56c-47d6-9a10-531506720261" containerName="payload-processing" containerID="cri-o://f7e101f3445b39a2e25006f17fa5749d8f15da09ef5a50f6cc965f28da85cb3b" gracePeriod=25 Oct 30 20:54:13.174124 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:13.174098 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6" Oct 30 20:54:13.272341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:13.272290 4848 generic.go:356] "Generic (PLEG): container finished" podID="05c28ea7-d56c-47d6-9a10-531506720261" containerID="f7e101f3445b39a2e25006f17fa5749d8f15da09ef5a50f6cc965f28da85cb3b" exitCode=0 Oct 30 20:54:13.272548 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:13.272375 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6" event={"ID":"05c28ea7-d56c-47d6-9a10-531506720261","Type":"ContainerDied","Data":"f7e101f3445b39a2e25006f17fa5749d8f15da09ef5a50f6cc965f28da85cb3b"} Oct 30 20:54:13.272548 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:13.272467 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6" event={"ID":"05c28ea7-d56c-47d6-9a10-531506720261","Type":"ContainerDied","Data":"897fe383c5a98658e98a6298789c3ec43c5b342000339b47baf9753c2b2991c6"} Oct 30 20:54:13.272548 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:13.272504 4848 scope.go:122] "RemoveContainer" containerID="f7e101f3445b39a2e25006f17fa5749d8f15da09ef5a50f6cc965f28da85cb3b" Oct 30 20:54:13.272738 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:13.272697 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6" Oct 30 20:54:13.288233 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:13.288207 4848 scope.go:122] "RemoveContainer" containerID="f7e101f3445b39a2e25006f17fa5749d8f15da09ef5a50f6cc965f28da85cb3b" Oct 30 20:54:13.288574 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:54:13.288552 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e101f3445b39a2e25006f17fa5749d8f15da09ef5a50f6cc965f28da85cb3b\": container with ID starting with f7e101f3445b39a2e25006f17fa5749d8f15da09ef5a50f6cc965f28da85cb3b not found: ID does not exist" containerID="f7e101f3445b39a2e25006f17fa5749d8f15da09ef5a50f6cc965f28da85cb3b" Oct 30 20:54:13.288637 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:13.288585 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e101f3445b39a2e25006f17fa5749d8f15da09ef5a50f6cc965f28da85cb3b"} err="failed to get container status \"f7e101f3445b39a2e25006f17fa5749d8f15da09ef5a50f6cc965f28da85cb3b\": rpc error: code = NotFound desc = could not find container \"f7e101f3445b39a2e25006f17fa5749d8f15da09ef5a50f6cc965f28da85cb3b\": container with ID starting with f7e101f3445b39a2e25006f17fa5749d8f15da09ef5a50f6cc965f28da85cb3b not found: ID does not exist" Oct 30 20:54:13.295758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:13.295733 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05c28ea7-d56c-47d6-9a10-531506720261-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c28ea7-d56c-47d6-9a10-531506720261-plugins-config-volume\") pod \"05c28ea7-d56c-47d6-9a10-531506720261\" (UID: \"05c28ea7-d56c-47d6-9a10-531506720261\") " Oct 30 20:54:13.295928 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:13.295915 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c28ea7-d56c-47d6-9a10-531506720261-kube-api-access-pgvhz\" (UniqueName: \"kubernetes.io/projected/05c28ea7-d56c-47d6-9a10-531506720261-kube-api-access-pgvhz\") pod \"05c28ea7-d56c-47d6-9a10-531506720261\" (UID: \"05c28ea7-d56c-47d6-9a10-531506720261\") " Oct 30 20:54:13.296112 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:13.296086 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c28ea7-d56c-47d6-9a10-531506720261-plugins-config-volume" pod "05c28ea7-d56c-47d6-9a10-531506720261" (UID: "05c28ea7-d56c-47d6-9a10-531506720261"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:13.296446 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:13.296424 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c28ea7-d56c-47d6-9a10-531506720261-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:13.297950 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:13.297927 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c28ea7-d56c-47d6-9a10-531506720261-kube-api-access-pgvhz" pod "05c28ea7-d56c-47d6-9a10-531506720261" (UID: "05c28ea7-d56c-47d6-9a10-531506720261"). InnerVolumeSpecName "kube-api-access-pgvhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:13.381249 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:13.381213 4848 patch_prober.go:29] interesting pod/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 20:54:13.381400 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:13.381279 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" podUID="d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 20:54:13.397693 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:13.397621 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgvhz\" (UniqueName: \"kubernetes.io/projected/05c28ea7-d56c-47d6-9a10-531506720261-kube-api-access-pgvhz\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:13.609901 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:13.609869 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6"] Oct 30 20:54:13.622910 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:13.622878 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1982ff-78b566b7c5-kjpv6"] Oct 30 20:54:14.385974 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:14.385946 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:54:14.386398 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:14.386371 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:54:14.387068 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:14.387051 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:54:14.934045 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:14.934017 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05c28ea7-d56c-47d6-9a10-531506720261" path="/var/lib/kubelet/pods/05c28ea7-d56c-47d6-9a10-531506720261/volumes" Oct 30 20:54:15.290331 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.290299 4848 generic.go:356] "Generic (PLEG): container finished" podID="a79a6860-1766-4d6b-9bb7-ccfa69a02989" containerID="accc45a41f4e009c3fd07a4d60498e68b1ba5081e65f4d4673fb823d010e4c37" exitCode=0 Oct 30 20:54:15.290464 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.290403 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" event={"ID":"a79a6860-1766-4d6b-9bb7-ccfa69a02989","Type":"ContainerDied","Data":"accc45a41f4e009c3fd07a4d60498e68b1ba5081e65f4d4673fb823d010e4c37"} Oct 30 20:54:15.290523 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.290477 4848 scope.go:122] "RemoveContainer" containerID="12f5e8040c139384dd8b5f1fea17ac812928b78f1d108401eca7208ac2caba64" Oct 30 20:54:15.436915 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.436890 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:54:15.539106 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.539020 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-podinfo\") pod \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " Oct 30 20:54:15.539259 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.539118 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-workload-socket\") pod \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " Oct 30 20:54:15.539305 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.539283 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a79a6860-1766-4d6b-9bb7-ccfa69a02989-kube-api-access-k4bgw\" (UniqueName: \"kubernetes.io/projected/a79a6860-1766-4d6b-9bb7-ccfa69a02989-kube-api-access-k4bgw\") pod \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " Oct 30 20:54:15.539430 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.539404 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-data\") pod \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " Oct 30 20:54:15.539540 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.539464 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istiod-ca-cert\") pod \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " Oct 30 20:54:15.539540 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.539501 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-token\" (UniqueName: \"kubernetes.io/projected/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-token\") pod \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " Oct 30 20:54:15.539540 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.539528 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-credential-socket\") pod \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " Oct 30 20:54:15.539737 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.539589 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-workload-certs\") pod \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " Oct 30 20:54:15.539737 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.539634 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-envoy\") pod \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\" (UID: \"a79a6860-1766-4d6b-9bb7-ccfa69a02989\") " Oct 30 20:54:15.539935 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.539913 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-data" pod "a79a6860-1766-4d6b-9bb7-ccfa69a02989" (UID: "a79a6860-1766-4d6b-9bb7-ccfa69a02989"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:15.540214 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.540197 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:15.540488 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.540444 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-credential-socket" pod "a79a6860-1766-4d6b-9bb7-ccfa69a02989" (UID: "a79a6860-1766-4d6b-9bb7-ccfa69a02989"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:15.540670 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.540636 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-workload-socket" pod "a79a6860-1766-4d6b-9bb7-ccfa69a02989" (UID: "a79a6860-1766-4d6b-9bb7-ccfa69a02989"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:15.542730 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.540966 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-workload-certs" pod "a79a6860-1766-4d6b-9bb7-ccfa69a02989" (UID: "a79a6860-1766-4d6b-9bb7-ccfa69a02989"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:15.542730 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.541450 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istiod-ca-cert" pod "a79a6860-1766-4d6b-9bb7-ccfa69a02989" (UID: "a79a6860-1766-4d6b-9bb7-ccfa69a02989"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:15.543271 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.543234 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-token" pod "a79a6860-1766-4d6b-9bb7-ccfa69a02989" (UID: "a79a6860-1766-4d6b-9bb7-ccfa69a02989"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:15.543496 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.543476 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79a6860-1766-4d6b-9bb7-ccfa69a02989-kube-api-access-k4bgw" pod "a79a6860-1766-4d6b-9bb7-ccfa69a02989" (UID: "a79a6860-1766-4d6b-9bb7-ccfa69a02989"). InnerVolumeSpecName "kube-api-access-k4bgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:15.543901 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.543876 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-envoy" pod "a79a6860-1766-4d6b-9bb7-ccfa69a02989" (UID: "a79a6860-1766-4d6b-9bb7-ccfa69a02989"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:15.544022 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.544003 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-podinfo" pod "a79a6860-1766-4d6b-9bb7-ccfa69a02989" (UID: "a79a6860-1766-4d6b-9bb7-ccfa69a02989"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:54:15.648618 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.648580 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:15.648618 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.648618 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:15.648915 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.648633 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:15.648915 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.648646 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:15.648915 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.648662 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:15.648915 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.648675 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a79a6860-1766-4d6b-9bb7-ccfa69a02989-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:15.648915 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.648688 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a79a6860-1766-4d6b-9bb7-ccfa69a02989-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:15.648915 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:15.648704 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4bgw\" (UniqueName: \"kubernetes.io/projected/a79a6860-1766-4d6b-9bb7-ccfa69a02989-kube-api-access-k4bgw\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:16.304299 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:16.299744 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" event={"ID":"a79a6860-1766-4d6b-9bb7-ccfa69a02989","Type":"ContainerDied","Data":"9f1c46e99a83c43b0d659268a92d509ba0dab83f6db27dc1b78640e2f6da3742"} Oct 30 20:54:16.304299 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:16.299865 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl" Oct 30 20:54:16.304299 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:16.299895 4848 scope.go:122] "RemoveContainer" containerID="accc45a41f4e009c3fd07a4d60498e68b1ba5081e65f4d4673fb823d010e4c37" Oct 30 20:54:16.330084 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:16.330055 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl"] Oct 30 20:54:16.334931 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:16.334905 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1fac16-openshift-default-cdc6dff7c-lq4sl"] Oct 30 20:54:16.933846 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:16.933815 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a79a6860-1766-4d6b-9bb7-ccfa69a02989" path="/var/lib/kubelet/pods/a79a6860-1766-4d6b-9bb7-ccfa69a02989/volumes" Oct 30 20:54:20.134796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:20.131273 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv"] Oct 30 20:54:20.134796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:20.131513 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" podUID="338bef5a-21a8-4a89-8c84-539bdb24482b" containerName="maas-api" containerID="cri-o://6edc7548dbb0570bb51e1413ad5c00c82bb372e44c4c456820161b94142cf956" gracePeriod=30 Oct 30 20:54:20.603429 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:20.599265 4848 generic.go:356] "Generic (PLEG): container finished" podID="338bef5a-21a8-4a89-8c84-539bdb24482b" containerID="6edc7548dbb0570bb51e1413ad5c00c82bb372e44c4c456820161b94142cf956" exitCode=0 Oct 30 20:54:20.604113 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:20.603813 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" event={"ID":"338bef5a-21a8-4a89-8c84-539bdb24482b","Type":"ContainerDied","Data":"6edc7548dbb0570bb51e1413ad5c00c82bb372e44c4c456820161b94142cf956"} Oct 30 20:54:20.733434 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:20.729024 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" Oct 30 20:54:20.796363 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:20.796319 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/338bef5a-21a8-4a89-8c84-539bdb24482b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/338bef5a-21a8-4a89-8c84-539bdb24482b-metrics-certs\") pod \"338bef5a-21a8-4a89-8c84-539bdb24482b\" (UID: \"338bef5a-21a8-4a89-8c84-539bdb24482b\") " Oct 30 20:54:20.796578 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:20.796541 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/338bef5a-21a8-4a89-8c84-539bdb24482b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/338bef5a-21a8-4a89-8c84-539bdb24482b-maas-api-tls\") pod \"338bef5a-21a8-4a89-8c84-539bdb24482b\" (UID: \"338bef5a-21a8-4a89-8c84-539bdb24482b\") " Oct 30 20:54:20.796687 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:20.796670 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/338bef5a-21a8-4a89-8c84-539bdb24482b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/338bef5a-21a8-4a89-8c84-539bdb24482b-openshift-service-ca\") pod \"338bef5a-21a8-4a89-8c84-539bdb24482b\" (UID: \"338bef5a-21a8-4a89-8c84-539bdb24482b\") " Oct 30 20:54:20.796959 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:20.796891 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/338bef5a-21a8-4a89-8c84-539bdb24482b-kube-api-access-q2xgz\" (UniqueName: \"kubernetes.io/projected/338bef5a-21a8-4a89-8c84-539bdb24482b-kube-api-access-q2xgz\") pod \"338bef5a-21a8-4a89-8c84-539bdb24482b\" (UID: \"338bef5a-21a8-4a89-8c84-539bdb24482b\") " Oct 30 20:54:20.806647 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:20.802018 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/338bef5a-21a8-4a89-8c84-539bdb24482b-openshift-service-ca" pod "338bef5a-21a8-4a89-8c84-539bdb24482b" (UID: "338bef5a-21a8-4a89-8c84-539bdb24482b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:20.812797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:20.810969 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/338bef5a-21a8-4a89-8c84-539bdb24482b-maas-api-tls" pod "338bef5a-21a8-4a89-8c84-539bdb24482b" (UID: "338bef5a-21a8-4a89-8c84-539bdb24482b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:20.816790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:20.812948 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/338bef5a-21a8-4a89-8c84-539bdb24482b-metrics-certs" pod "338bef5a-21a8-4a89-8c84-539bdb24482b" (UID: "338bef5a-21a8-4a89-8c84-539bdb24482b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:20.834828 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:20.820277 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/338bef5a-21a8-4a89-8c84-539bdb24482b-kube-api-access-q2xgz" pod "338bef5a-21a8-4a89-8c84-539bdb24482b" (UID: "338bef5a-21a8-4a89-8c84-539bdb24482b"). InnerVolumeSpecName "kube-api-access-q2xgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:20.904003 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:20.902508 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/338bef5a-21a8-4a89-8c84-539bdb24482b-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:20.904003 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:20.902538 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2xgz\" (UniqueName: \"kubernetes.io/projected/338bef5a-21a8-4a89-8c84-539bdb24482b-kube-api-access-q2xgz\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:20.904003 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:20.902552 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/338bef5a-21a8-4a89-8c84-539bdb24482b-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:20.904003 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:20.902565 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/338bef5a-21a8-4a89-8c84-539bdb24482b-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:21.035736 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:21.035704 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z"] Oct 30 20:54:21.612491 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:21.612460 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" event={"ID":"338bef5a-21a8-4a89-8c84-539bdb24482b","Type":"ContainerDied","Data":"251fc6d33d791dcb332f13b3049424f4cea36f2f30136c31aa6a248da7eeee84"} Oct 30 20:54:21.612935 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:21.612507 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv" Oct 30 20:54:21.612935 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:21.612522 4848 scope.go:122] "RemoveContainer" containerID="6edc7548dbb0570bb51e1413ad5c00c82bb372e44c4c456820161b94142cf956" Oct 30 20:54:21.654757 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:21.654729 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv"] Oct 30 20:54:21.677181 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:21.677157 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-95a684-79846bb7f8-5zqnv"] Oct 30 20:54:22.510493 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:54:22.510455 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:54:22.935753 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:22.935721 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="338bef5a-21a8-4a89-8c84-539bdb24482b" path="/var/lib/kubelet/pods/338bef5a-21a8-4a89-8c84-539bdb24482b/volumes" Oct 30 20:54:26.028477 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:26.028448 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8"] Oct 30 20:54:26.036902 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:26.036867 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z" podUID="f616d8f8-b034-4199-8bb0-fe3999e85785" containerName="payload-pre-processing" containerID="cri-o://bcc74a62eb2f96454a9a99b25b2b7f3771ad0d671cfe5e4a3a152d743b0b9ffa" gracePeriod=25 Oct 30 20:54:26.050001 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:26.049971 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj"] Oct 30 20:54:26.411487 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:26.411451 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z" Oct 30 20:54:26.483000 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:26.482969 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f616d8f8-b034-4199-8bb0-fe3999e85785-kube-api-access-bcxhq\" (UniqueName: \"kubernetes.io/projected/f616d8f8-b034-4199-8bb0-fe3999e85785-kube-api-access-bcxhq\") pod \"f616d8f8-b034-4199-8bb0-fe3999e85785\" (UID: \"f616d8f8-b034-4199-8bb0-fe3999e85785\") " Oct 30 20:54:26.483568 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:26.483530 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f616d8f8-b034-4199-8bb0-fe3999e85785-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f616d8f8-b034-4199-8bb0-fe3999e85785-plugins-config-volume\") pod \"f616d8f8-b034-4199-8bb0-fe3999e85785\" (UID: \"f616d8f8-b034-4199-8bb0-fe3999e85785\") " Oct 30 20:54:26.485298 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:26.484315 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f616d8f8-b034-4199-8bb0-fe3999e85785-plugins-config-volume" pod "f616d8f8-b034-4199-8bb0-fe3999e85785" (UID: "f616d8f8-b034-4199-8bb0-fe3999e85785"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:26.485928 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:26.485737 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f616d8f8-b034-4199-8bb0-fe3999e85785-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:26.611148 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:26.611047 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f616d8f8-b034-4199-8bb0-fe3999e85785-kube-api-access-bcxhq" pod "f616d8f8-b034-4199-8bb0-fe3999e85785" (UID: "f616d8f8-b034-4199-8bb0-fe3999e85785"). InnerVolumeSpecName "kube-api-access-bcxhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:26.648828 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:26.648792 4848 generic.go:356] "Generic (PLEG): container finished" podID="f616d8f8-b034-4199-8bb0-fe3999e85785" containerID="bcc74a62eb2f96454a9a99b25b2b7f3771ad0d671cfe5e4a3a152d743b0b9ffa" exitCode=0 Oct 30 20:54:26.648980 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:26.648839 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z" Oct 30 20:54:26.648980 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:26.648878 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z" event={"ID":"f616d8f8-b034-4199-8bb0-fe3999e85785","Type":"ContainerDied","Data":"bcc74a62eb2f96454a9a99b25b2b7f3771ad0d671cfe5e4a3a152d743b0b9ffa"} Oct 30 20:54:26.648980 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:26.648931 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z" event={"ID":"f616d8f8-b034-4199-8bb0-fe3999e85785","Type":"ContainerDied","Data":"4f40b13efbc6e3cd48a45d8d66247d07c3464a5d420b07ba5065b313ef5657e5"} Oct 30 20:54:26.648980 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:26.648953 4848 scope.go:122] "RemoveContainer" containerID="bcc74a62eb2f96454a9a99b25b2b7f3771ad0d671cfe5e4a3a152d743b0b9ffa" Oct 30 20:54:26.659904 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:26.659888 4848 scope.go:122] "RemoveContainer" containerID="bcc74a62eb2f96454a9a99b25b2b7f3771ad0d671cfe5e4a3a152d743b0b9ffa" Oct 30 20:54:26.660164 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:54:26.660139 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcc74a62eb2f96454a9a99b25b2b7f3771ad0d671cfe5e4a3a152d743b0b9ffa\": container with ID starting with bcc74a62eb2f96454a9a99b25b2b7f3771ad0d671cfe5e4a3a152d743b0b9ffa not found: ID does not exist" containerID="bcc74a62eb2f96454a9a99b25b2b7f3771ad0d671cfe5e4a3a152d743b0b9ffa" Oct 30 20:54:26.660285 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:26.660172 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcc74a62eb2f96454a9a99b25b2b7f3771ad0d671cfe5e4a3a152d743b0b9ffa"} err="failed to get container status \"bcc74a62eb2f96454a9a99b25b2b7f3771ad0d671cfe5e4a3a152d743b0b9ffa\": rpc error: code = NotFound desc = could not find container \"bcc74a62eb2f96454a9a99b25b2b7f3771ad0d671cfe5e4a3a152d743b0b9ffa\": container with ID starting with bcc74a62eb2f96454a9a99b25b2b7f3771ad0d671cfe5e4a3a152d743b0b9ffa not found: ID does not exist" Oct 30 20:54:26.686630 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:26.686607 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z"] Oct 30 20:54:26.704816 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:26.701243 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcxhq\" (UniqueName: \"kubernetes.io/projected/f616d8f8-b034-4199-8bb0-fe3999e85785-kube-api-access-bcxhq\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:26.737337 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:26.737086 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-49990a-bb6cd55dc-d7c4z"] Oct 30 20:54:26.932782 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:26.932739 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f616d8f8-b034-4199-8bb0-fe3999e85785" path="/var/lib/kubelet/pods/f616d8f8-b034-4199-8bb0-fe3999e85785/volumes" Oct 30 20:54:31.031962 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:31.031915 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8" podUID="e029fa76-d526-498e-acbd-efda2b093887" containerName="payload-processing" containerID="cri-o://8823367d236c4ea9dfa1cf69adb3bf1886f17c4b8d7fa6e68ca4b343e7bde1a5" gracePeriod=25 Oct 30 20:54:31.059451 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:31.056838 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj" podUID="c3f2d488-7956-4a7a-8b44-53eabadd98bc" containerName="payload-pre-processing" containerID="cri-o://a6caef4dce9bdad6e10a6bc0f6a25afab8d63beafab4412349881c8074b4296d" gracePeriod=25 Oct 30 20:54:32.576951 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:32.576246 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c"] Oct 30 20:54:32.576951 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:32.576554 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" podUID="6c39353d-db8a-4650-bf95-3ca9bc3be519" containerName="maas-api" containerID="cri-o://5a3631b5718a5f476496d3825d94829a9d65db2b3dcd28dd5e69cf362f4aaa72" gracePeriod=30 Oct 30 20:54:32.742106 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:32.742077 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq"] Oct 30 20:54:32.742826 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:32.742386 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" podUID="3199622d-dbd3-4e39-8376-93322d66b0ae" containerName="istio-proxy" containerID="cri-o://b52c654dd6100dd368a83aef50156141cd132e54710968714399cc19cde86289" gracePeriod=30 Oct 30 20:54:32.777068 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:32.777034 4848 generic.go:356] "Generic (PLEG): container finished" podID="6c39353d-db8a-4650-bf95-3ca9bc3be519" containerID="5a3631b5718a5f476496d3825d94829a9d65db2b3dcd28dd5e69cf362f4aaa72" exitCode=0 Oct 30 20:54:32.777284 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:32.777265 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" event={"ID":"6c39353d-db8a-4650-bf95-3ca9bc3be519","Type":"ContainerDied","Data":"5a3631b5718a5f476496d3825d94829a9d65db2b3dcd28dd5e69cf362f4aaa72"} Oct 30 20:54:33.233078 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:33.232935 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" Oct 30 20:54:33.257166 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:33.257108 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c39353d-db8a-4650-bf95-3ca9bc3be519-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c39353d-db8a-4650-bf95-3ca9bc3be519-metrics-certs\") pod \"6c39353d-db8a-4650-bf95-3ca9bc3be519\" (UID: \"6c39353d-db8a-4650-bf95-3ca9bc3be519\") " Oct 30 20:54:33.257826 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:33.257800 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c39353d-db8a-4650-bf95-3ca9bc3be519-kube-api-access-lnffp\" (UniqueName: \"kubernetes.io/projected/6c39353d-db8a-4650-bf95-3ca9bc3be519-kube-api-access-lnffp\") pod \"6c39353d-db8a-4650-bf95-3ca9bc3be519\" (UID: \"6c39353d-db8a-4650-bf95-3ca9bc3be519\") " Oct 30 20:54:33.258049 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:33.258032 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c39353d-db8a-4650-bf95-3ca9bc3be519-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c39353d-db8a-4650-bf95-3ca9bc3be519-maas-api-tls\") pod \"6c39353d-db8a-4650-bf95-3ca9bc3be519\" (UID: \"6c39353d-db8a-4650-bf95-3ca9bc3be519\") " Oct 30 20:54:33.258235 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:33.258220 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c39353d-db8a-4650-bf95-3ca9bc3be519-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c39353d-db8a-4650-bf95-3ca9bc3be519-openshift-service-ca\") pod \"6c39353d-db8a-4650-bf95-3ca9bc3be519\" (UID: \"6c39353d-db8a-4650-bf95-3ca9bc3be519\") " Oct 30 20:54:33.260276 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:33.258821 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c39353d-db8a-4650-bf95-3ca9bc3be519-openshift-service-ca" pod "6c39353d-db8a-4650-bf95-3ca9bc3be519" (UID: "6c39353d-db8a-4650-bf95-3ca9bc3be519"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:33.275372 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:33.275317 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c39353d-db8a-4650-bf95-3ca9bc3be519-metrics-certs" pod "6c39353d-db8a-4650-bf95-3ca9bc3be519" (UID: "6c39353d-db8a-4650-bf95-3ca9bc3be519"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:33.277264 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:33.277231 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c39353d-db8a-4650-bf95-3ca9bc3be519-maas-api-tls" pod "6c39353d-db8a-4650-bf95-3ca9bc3be519" (UID: "6c39353d-db8a-4650-bf95-3ca9bc3be519"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:33.278089 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:33.278041 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c39353d-db8a-4650-bf95-3ca9bc3be519-kube-api-access-lnffp" pod "6c39353d-db8a-4650-bf95-3ca9bc3be519" (UID: "6c39353d-db8a-4650-bf95-3ca9bc3be519"). InnerVolumeSpecName "kube-api-access-lnffp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:33.398058 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:33.390340 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnffp\" (UniqueName: \"kubernetes.io/projected/6c39353d-db8a-4650-bf95-3ca9bc3be519-kube-api-access-lnffp\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:33.398058 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:33.390376 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c39353d-db8a-4650-bf95-3ca9bc3be519-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:33.398058 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:33.390392 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c39353d-db8a-4650-bf95-3ca9bc3be519-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:33.398058 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:33.390406 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c39353d-db8a-4650-bf95-3ca9bc3be519-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:33.785804 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:33.785753 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" event={"ID":"6c39353d-db8a-4650-bf95-3ca9bc3be519","Type":"ContainerDied","Data":"7ae44e899fcb57a9a5d9769b8bbce69961fa40eb13ddbf7f6062027319cab190"} Oct 30 20:54:33.785804 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:33.785791 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c" Oct 30 20:54:33.786316 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:33.785865 4848 scope.go:122] "RemoveContainer" containerID="5a3631b5718a5f476496d3825d94829a9d65db2b3dcd28dd5e69cf362f4aaa72" Oct 30 20:54:33.809924 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:33.809898 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c"] Oct 30 20:54:33.815467 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:33.815444 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-49990a-57b6b89458-lh42c"] Oct 30 20:54:34.933230 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:34.933199 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c39353d-db8a-4650-bf95-3ca9bc3be519" path="/var/lib/kubelet/pods/6c39353d-db8a-4650-bf95-3ca9bc3be519/volumes" Oct 30 20:54:38.019074 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.018985 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj" podUID="c3f2d488-7956-4a7a-8b44-53eabadd98bc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:54:38.019617 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.019305 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8" podUID="e029fa76-d526-498e-acbd-efda2b093887" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:54:38.065835 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.065811 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:54:38.195562 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.195532 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-token\" (UniqueName: \"kubernetes.io/projected/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-token\") pod \"3199622d-dbd3-4e39-8376-93322d66b0ae\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " Oct 30 20:54:38.195748 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.195571 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3199622d-dbd3-4e39-8376-93322d66b0ae-kube-api-access-ggmxj\" (UniqueName: \"kubernetes.io/projected/3199622d-dbd3-4e39-8376-93322d66b0ae-kube-api-access-ggmxj\") pod \"3199622d-dbd3-4e39-8376-93322d66b0ae\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " Oct 30 20:54:38.195748 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.195617 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-data\") pod \"3199622d-dbd3-4e39-8376-93322d66b0ae\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " Oct 30 20:54:38.195748 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.195639 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-podinfo\") pod \"3199622d-dbd3-4e39-8376-93322d66b0ae\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " Oct 30 20:54:38.195748 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.195704 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-envoy\") pod \"3199622d-dbd3-4e39-8376-93322d66b0ae\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " Oct 30 20:54:38.195998 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.195867 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-credential-socket\") pod \"3199622d-dbd3-4e39-8376-93322d66b0ae\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " Oct 30 20:54:38.195998 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.195896 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-workload-socket\") pod \"3199622d-dbd3-4e39-8376-93322d66b0ae\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " Oct 30 20:54:38.195998 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.195960 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-data" pod "3199622d-dbd3-4e39-8376-93322d66b0ae" (UID: "3199622d-dbd3-4e39-8376-93322d66b0ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:38.196127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.196090 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-workload-certs\") pod \"3199622d-dbd3-4e39-8376-93322d66b0ae\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " Oct 30 20:54:38.196179 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.196145 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3199622d-dbd3-4e39-8376-93322d66b0ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3199622d-dbd3-4e39-8376-93322d66b0ae-istiod-ca-cert\") pod \"3199622d-dbd3-4e39-8376-93322d66b0ae\" (UID: \"3199622d-dbd3-4e39-8376-93322d66b0ae\") " Oct 30 20:54:38.196701 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.196480 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-credential-socket" pod "3199622d-dbd3-4e39-8376-93322d66b0ae" (UID: "3199622d-dbd3-4e39-8376-93322d66b0ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:38.196850 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.196755 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-workload-socket" pod "3199622d-dbd3-4e39-8376-93322d66b0ae" (UID: "3199622d-dbd3-4e39-8376-93322d66b0ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:38.196850 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.196753 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-workload-certs" pod "3199622d-dbd3-4e39-8376-93322d66b0ae" (UID: "3199622d-dbd3-4e39-8376-93322d66b0ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:38.196985 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.196849 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3199622d-dbd3-4e39-8376-93322d66b0ae-istiod-ca-cert" pod "3199622d-dbd3-4e39-8376-93322d66b0ae" (UID: "3199622d-dbd3-4e39-8376-93322d66b0ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:38.196985 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.196961 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:38.196985 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.196978 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3199622d-dbd3-4e39-8376-93322d66b0ae-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:38.197149 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.196988 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:38.197149 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.197002 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:38.197149 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.197015 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:38.198344 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.198317 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-token" pod "3199622d-dbd3-4e39-8376-93322d66b0ae" (UID: "3199622d-dbd3-4e39-8376-93322d66b0ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:38.198461 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.198345 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3199622d-dbd3-4e39-8376-93322d66b0ae-kube-api-access-ggmxj" pod "3199622d-dbd3-4e39-8376-93322d66b0ae" (UID: "3199622d-dbd3-4e39-8376-93322d66b0ae"). InnerVolumeSpecName "kube-api-access-ggmxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:38.198512 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.198482 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-envoy" pod "3199622d-dbd3-4e39-8376-93322d66b0ae" (UID: "3199622d-dbd3-4e39-8376-93322d66b0ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:54:38.199492 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.199470 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-podinfo" pod "3199622d-dbd3-4e39-8376-93322d66b0ae" (UID: "3199622d-dbd3-4e39-8376-93322d66b0ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:54:38.298488 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.298416 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:38.298488 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.298443 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggmxj\" (UniqueName: \"kubernetes.io/projected/3199622d-dbd3-4e39-8376-93322d66b0ae-kube-api-access-ggmxj\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:38.298488 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.298454 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:38.298488 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.298464 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3199622d-dbd3-4e39-8376-93322d66b0ae-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:38.845836 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.845801 4848 generic.go:356] "Generic (PLEG): container finished" podID="3199622d-dbd3-4e39-8376-93322d66b0ae" containerID="b52c654dd6100dd368a83aef50156141cd132e54710968714399cc19cde86289" exitCode=0 Oct 30 20:54:38.846059 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.845856 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" Oct 30 20:54:38.846059 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.845884 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" event={"ID":"3199622d-dbd3-4e39-8376-93322d66b0ae","Type":"ContainerDied","Data":"b52c654dd6100dd368a83aef50156141cd132e54710968714399cc19cde86289"} Oct 30 20:54:38.846059 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.845965 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq" event={"ID":"3199622d-dbd3-4e39-8376-93322d66b0ae","Type":"ContainerDied","Data":"f2abea9941b97d7daf92a991f0f425e48403ded93e40c8fdf108598c8550cab1"} Oct 30 20:54:38.846059 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.845989 4848 scope.go:122] "RemoveContainer" containerID="b52c654dd6100dd368a83aef50156141cd132e54710968714399cc19cde86289" Oct 30 20:54:38.857350 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.857333 4848 scope.go:122] "RemoveContainer" containerID="4c9a76b6be18f39f2948f7604574f1a3a5288c89ff0aa9225eb3cb2049a6fd34" Oct 30 20:54:38.867984 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.867967 4848 scope.go:122] "RemoveContainer" containerID="b52c654dd6100dd368a83aef50156141cd132e54710968714399cc19cde86289" Oct 30 20:54:38.868294 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:54:38.868267 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b52c654dd6100dd368a83aef50156141cd132e54710968714399cc19cde86289\": container with ID starting with b52c654dd6100dd368a83aef50156141cd132e54710968714399cc19cde86289 not found: ID does not exist" containerID="b52c654dd6100dd368a83aef50156141cd132e54710968714399cc19cde86289" Oct 30 20:54:38.868371 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.868306 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b52c654dd6100dd368a83aef50156141cd132e54710968714399cc19cde86289"} err="failed to get container status \"b52c654dd6100dd368a83aef50156141cd132e54710968714399cc19cde86289\": rpc error: code = NotFound desc = could not find container \"b52c654dd6100dd368a83aef50156141cd132e54710968714399cc19cde86289\": container with ID starting with b52c654dd6100dd368a83aef50156141cd132e54710968714399cc19cde86289 not found: ID does not exist" Oct 30 20:54:38.868371 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.868337 4848 scope.go:122] "RemoveContainer" containerID="4c9a76b6be18f39f2948f7604574f1a3a5288c89ff0aa9225eb3cb2049a6fd34" Oct 30 20:54:38.868627 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:54:38.868604 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c9a76b6be18f39f2948f7604574f1a3a5288c89ff0aa9225eb3cb2049a6fd34\": container with ID starting with 4c9a76b6be18f39f2948f7604574f1a3a5288c89ff0aa9225eb3cb2049a6fd34 not found: ID does not exist" containerID="4c9a76b6be18f39f2948f7604574f1a3a5288c89ff0aa9225eb3cb2049a6fd34" Oct 30 20:54:38.868701 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.868637 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c9a76b6be18f39f2948f7604574f1a3a5288c89ff0aa9225eb3cb2049a6fd34"} err="failed to get container status \"4c9a76b6be18f39f2948f7604574f1a3a5288c89ff0aa9225eb3cb2049a6fd34\": rpc error: code = NotFound desc = could not find container \"4c9a76b6be18f39f2948f7604574f1a3a5288c89ff0aa9225eb3cb2049a6fd34\": container with ID starting with 4c9a76b6be18f39f2948f7604574f1a3a5288c89ff0aa9225eb3cb2049a6fd34 not found: ID does not exist" Oct 30 20:54:38.876378 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.876359 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq"] Oct 30 20:54:38.884760 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.884740 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1982ff-openshift-default-d7d5775d9-gsmhq"] Oct 30 20:54:38.933461 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:38.933424 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3199622d-dbd3-4e39-8376-93322d66b0ae" path="/var/lib/kubelet/pods/3199622d-dbd3-4e39-8376-93322d66b0ae/volumes" Oct 30 20:54:48.029169 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:48.026850 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj" podUID="c3f2d488-7956-4a7a-8b44-53eabadd98bc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:54:48.029689 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:48.029637 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8" podUID="e029fa76-d526-498e-acbd-efda2b093887" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:54:49.391974 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:49.389366 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw"] Oct 30 20:54:49.391974 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:49.389696 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" podUID="e78c510b-f4e1-4d65-93cc-2b00f0896b81" containerName="maas-api" containerID="cri-o://92b953df46ce805af56d91f8103d5d0f9e08d386e2985c54d67fa6b404ef87fd" gracePeriod=30 Oct 30 20:54:49.954063 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:49.953950 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" Oct 30 20:54:49.985148 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:49.985112 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" Oct 30 20:54:49.985339 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:49.985114 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" event={"ID":"e78c510b-f4e1-4d65-93cc-2b00f0896b81","Type":"ContainerDied","Data":"92b953df46ce805af56d91f8103d5d0f9e08d386e2985c54d67fa6b404ef87fd"} Oct 30 20:54:49.985568 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:49.985039 4848 generic.go:356] "Generic (PLEG): container finished" podID="e78c510b-f4e1-4d65-93cc-2b00f0896b81" containerID="92b953df46ce805af56d91f8103d5d0f9e08d386e2985c54d67fa6b404ef87fd" exitCode=0 Oct 30 20:54:49.985802 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:49.985759 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw" event={"ID":"e78c510b-f4e1-4d65-93cc-2b00f0896b81","Type":"ContainerDied","Data":"cd3d67682e3ecbdb4fab99656022bb358d8d15a7e7aec4e98862d79f274e1307"} Oct 30 20:54:49.985903 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:49.985821 4848 scope.go:122] "RemoveContainer" containerID="92b953df46ce805af56d91f8103d5d0f9e08d386e2985c54d67fa6b404ef87fd" Oct 30 20:54:49.998118 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:49.998093 4848 scope.go:122] "RemoveContainer" containerID="92b953df46ce805af56d91f8103d5d0f9e08d386e2985c54d67fa6b404ef87fd" Oct 30 20:54:49.998508 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:54:49.998466 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b953df46ce805af56d91f8103d5d0f9e08d386e2985c54d67fa6b404ef87fd\": container with ID starting with 92b953df46ce805af56d91f8103d5d0f9e08d386e2985c54d67fa6b404ef87fd not found: ID does not exist" containerID="92b953df46ce805af56d91f8103d5d0f9e08d386e2985c54d67fa6b404ef87fd" Oct 30 20:54:49.998577 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:49.998506 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b953df46ce805af56d91f8103d5d0f9e08d386e2985c54d67fa6b404ef87fd"} err="failed to get container status \"92b953df46ce805af56d91f8103d5d0f9e08d386e2985c54d67fa6b404ef87fd\": rpc error: code = NotFound desc = could not find container \"92b953df46ce805af56d91f8103d5d0f9e08d386e2985c54d67fa6b404ef87fd\": container with ID starting with 92b953df46ce805af56d91f8103d5d0f9e08d386e2985c54d67fa6b404ef87fd not found: ID does not exist" Oct 30 20:54:50.043164 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:50.043130 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78c510b-f4e1-4d65-93cc-2b00f0896b81-kube-api-access-7qtjj\" (UniqueName: \"kubernetes.io/projected/e78c510b-f4e1-4d65-93cc-2b00f0896b81-kube-api-access-7qtjj\") pod \"e78c510b-f4e1-4d65-93cc-2b00f0896b81\" (UID: \"e78c510b-f4e1-4d65-93cc-2b00f0896b81\") " Oct 30 20:54:50.043355 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:50.043232 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e78c510b-f4e1-4d65-93cc-2b00f0896b81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e78c510b-f4e1-4d65-93cc-2b00f0896b81-metrics-certs\") pod \"e78c510b-f4e1-4d65-93cc-2b00f0896b81\" (UID: \"e78c510b-f4e1-4d65-93cc-2b00f0896b81\") " Oct 30 20:54:50.043355 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:50.043348 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e78c510b-f4e1-4d65-93cc-2b00f0896b81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e78c510b-f4e1-4d65-93cc-2b00f0896b81-maas-api-tls\") pod \"e78c510b-f4e1-4d65-93cc-2b00f0896b81\" (UID: \"e78c510b-f4e1-4d65-93cc-2b00f0896b81\") " Oct 30 20:54:50.043487 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:50.043381 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e78c510b-f4e1-4d65-93cc-2b00f0896b81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78c510b-f4e1-4d65-93cc-2b00f0896b81-openshift-service-ca\") pod \"e78c510b-f4e1-4d65-93cc-2b00f0896b81\" (UID: \"e78c510b-f4e1-4d65-93cc-2b00f0896b81\") " Oct 30 20:54:50.044128 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:50.043867 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78c510b-f4e1-4d65-93cc-2b00f0896b81-openshift-service-ca" pod "e78c510b-f4e1-4d65-93cc-2b00f0896b81" (UID: "e78c510b-f4e1-4d65-93cc-2b00f0896b81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:50.044128 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:50.044099 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78c510b-f4e1-4d65-93cc-2b00f0896b81-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:50.045903 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:50.045869 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e78c510b-f4e1-4d65-93cc-2b00f0896b81-metrics-certs" pod "e78c510b-f4e1-4d65-93cc-2b00f0896b81" (UID: "e78c510b-f4e1-4d65-93cc-2b00f0896b81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:50.045984 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:50.045953 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78c510b-f4e1-4d65-93cc-2b00f0896b81-kube-api-access-7qtjj" pod "e78c510b-f4e1-4d65-93cc-2b00f0896b81" (UID: "e78c510b-f4e1-4d65-93cc-2b00f0896b81"). InnerVolumeSpecName "kube-api-access-7qtjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:50.046733 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:50.046707 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e78c510b-f4e1-4d65-93cc-2b00f0896b81-maas-api-tls" pod "e78c510b-f4e1-4d65-93cc-2b00f0896b81" (UID: "e78c510b-f4e1-4d65-93cc-2b00f0896b81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:50.166799 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:50.155020 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qtjj\" (UniqueName: \"kubernetes.io/projected/e78c510b-f4e1-4d65-93cc-2b00f0896b81-kube-api-access-7qtjj\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:50.166799 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:50.155062 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e78c510b-f4e1-4d65-93cc-2b00f0896b81-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:50.166799 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:50.155077 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e78c510b-f4e1-4d65-93cc-2b00f0896b81-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:50.327328 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:50.327298 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw"] Oct 30 20:54:50.333998 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:50.333973 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-v9qbw"] Oct 30 20:54:50.933446 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:50.933413 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e78c510b-f4e1-4d65-93cc-2b00f0896b81" path="/var/lib/kubelet/pods/e78c510b-f4e1-4d65-93cc-2b00f0896b81/volumes" Oct 30 20:54:51.319061 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:51.319030 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8" Oct 30 20:54:51.378664 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:51.378640 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj" Oct 30 20:54:51.385819 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:51.385759 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e029fa76-d526-498e-acbd-efda2b093887-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e029fa76-d526-498e-acbd-efda2b093887-plugins-config-volume\") pod \"e029fa76-d526-498e-acbd-efda2b093887\" (UID: \"e029fa76-d526-498e-acbd-efda2b093887\") " Oct 30 20:54:51.386069 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:51.386037 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e029fa76-d526-498e-acbd-efda2b093887-kube-api-access-d5wv5\" (UniqueName: \"kubernetes.io/projected/e029fa76-d526-498e-acbd-efda2b093887-kube-api-access-d5wv5\") pod \"e029fa76-d526-498e-acbd-efda2b093887\" (UID: \"e029fa76-d526-498e-acbd-efda2b093887\") " Oct 30 20:54:51.386189 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:51.386101 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e029fa76-d526-498e-acbd-efda2b093887-plugins-config-volume" pod "e029fa76-d526-498e-acbd-efda2b093887" (UID: "e029fa76-d526-498e-acbd-efda2b093887"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:51.386490 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:51.386471 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e029fa76-d526-498e-acbd-efda2b093887-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:51.388184 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:51.388157 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e029fa76-d526-498e-acbd-efda2b093887-kube-api-access-d5wv5" pod "e029fa76-d526-498e-acbd-efda2b093887" (UID: "e029fa76-d526-498e-acbd-efda2b093887"). InnerVolumeSpecName "kube-api-access-d5wv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:51.487437 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:51.487341 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3f2d488-7956-4a7a-8b44-53eabadd98bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f2d488-7956-4a7a-8b44-53eabadd98bc-plugins-config-volume\") pod \"c3f2d488-7956-4a7a-8b44-53eabadd98bc\" (UID: \"c3f2d488-7956-4a7a-8b44-53eabadd98bc\") " Oct 30 20:54:51.487633 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:51.487615 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3f2d488-7956-4a7a-8b44-53eabadd98bc-kube-api-access-bdlrp\" (UniqueName: \"kubernetes.io/projected/c3f2d488-7956-4a7a-8b44-53eabadd98bc-kube-api-access-bdlrp\") pod \"c3f2d488-7956-4a7a-8b44-53eabadd98bc\" (UID: \"c3f2d488-7956-4a7a-8b44-53eabadd98bc\") " Oct 30 20:54:51.487718 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:51.487696 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3f2d488-7956-4a7a-8b44-53eabadd98bc-plugins-config-volume" pod "c3f2d488-7956-4a7a-8b44-53eabadd98bc" (UID: "c3f2d488-7956-4a7a-8b44-53eabadd98bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:51.488248 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:51.488228 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3f2d488-7956-4a7a-8b44-53eabadd98bc-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:51.488370 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:51.488251 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5wv5\" (UniqueName: \"kubernetes.io/projected/e029fa76-d526-498e-acbd-efda2b093887-kube-api-access-d5wv5\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:51.489715 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:51.489689 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f2d488-7956-4a7a-8b44-53eabadd98bc-kube-api-access-bdlrp" pod "c3f2d488-7956-4a7a-8b44-53eabadd98bc" (UID: "c3f2d488-7956-4a7a-8b44-53eabadd98bc"). InnerVolumeSpecName "kube-api-access-bdlrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:51.589912 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:51.589879 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdlrp\" (UniqueName: \"kubernetes.io/projected/c3f2d488-7956-4a7a-8b44-53eabadd98bc-kube-api-access-bdlrp\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:52.001361 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:52.001324 4848 generic.go:356] "Generic (PLEG): container finished" podID="e029fa76-d526-498e-acbd-efda2b093887" containerID="8823367d236c4ea9dfa1cf69adb3bf1886f17c4b8d7fa6e68ca4b343e7bde1a5" exitCode=0 Oct 30 20:54:52.001839 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:52.001368 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8" Oct 30 20:54:52.001839 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:52.001409 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8" event={"ID":"e029fa76-d526-498e-acbd-efda2b093887","Type":"ContainerDied","Data":"8823367d236c4ea9dfa1cf69adb3bf1886f17c4b8d7fa6e68ca4b343e7bde1a5"} Oct 30 20:54:52.001839 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:52.001459 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8" event={"ID":"e029fa76-d526-498e-acbd-efda2b093887","Type":"ContainerDied","Data":"c8e87100442cfdd6eba885ecfa71df182f16d45a8e25eaa9660314bb31b74e7b"} Oct 30 20:54:52.001839 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:52.001510 4848 scope.go:122] "RemoveContainer" containerID="8823367d236c4ea9dfa1cf69adb3bf1886f17c4b8d7fa6e68ca4b343e7bde1a5" Oct 30 20:54:52.003198 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:52.003174 4848 generic.go:356] "Generic (PLEG): container finished" podID="c3f2d488-7956-4a7a-8b44-53eabadd98bc" containerID="a6caef4dce9bdad6e10a6bc0f6a25afab8d63beafab4412349881c8074b4296d" exitCode=0 Oct 30 20:54:52.003308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:52.003212 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj" Oct 30 20:54:52.003308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:52.003219 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj" event={"ID":"c3f2d488-7956-4a7a-8b44-53eabadd98bc","Type":"ContainerDied","Data":"a6caef4dce9bdad6e10a6bc0f6a25afab8d63beafab4412349881c8074b4296d"} Oct 30 20:54:52.003308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:52.003258 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj" event={"ID":"c3f2d488-7956-4a7a-8b44-53eabadd98bc","Type":"ContainerDied","Data":"7fa062dad33ac66392b86786d8600db71513e0234c066a6bd77aae33c4b220fc"} Oct 30 20:54:52.013127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:52.013109 4848 scope.go:122] "RemoveContainer" containerID="8823367d236c4ea9dfa1cf69adb3bf1886f17c4b8d7fa6e68ca4b343e7bde1a5" Oct 30 20:54:52.013563 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:54:52.013532 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8823367d236c4ea9dfa1cf69adb3bf1886f17c4b8d7fa6e68ca4b343e7bde1a5\": container with ID starting with 8823367d236c4ea9dfa1cf69adb3bf1886f17c4b8d7fa6e68ca4b343e7bde1a5 not found: ID does not exist" containerID="8823367d236c4ea9dfa1cf69adb3bf1886f17c4b8d7fa6e68ca4b343e7bde1a5" Oct 30 20:54:52.013638 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:52.013574 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8823367d236c4ea9dfa1cf69adb3bf1886f17c4b8d7fa6e68ca4b343e7bde1a5"} err="failed to get container status \"8823367d236c4ea9dfa1cf69adb3bf1886f17c4b8d7fa6e68ca4b343e7bde1a5\": rpc error: code = NotFound desc = could not find container \"8823367d236c4ea9dfa1cf69adb3bf1886f17c4b8d7fa6e68ca4b343e7bde1a5\": container with ID starting with 8823367d236c4ea9dfa1cf69adb3bf1886f17c4b8d7fa6e68ca4b343e7bde1a5 not found: ID does not exist" Oct 30 20:54:52.013638 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:52.013592 4848 scope.go:122] "RemoveContainer" containerID="a6caef4dce9bdad6e10a6bc0f6a25afab8d63beafab4412349881c8074b4296d" Oct 30 20:54:52.025173 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:52.025154 4848 scope.go:122] "RemoveContainer" containerID="a6caef4dce9bdad6e10a6bc0f6a25afab8d63beafab4412349881c8074b4296d" Oct 30 20:54:52.025447 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:54:52.025429 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6caef4dce9bdad6e10a6bc0f6a25afab8d63beafab4412349881c8074b4296d\": container with ID starting with a6caef4dce9bdad6e10a6bc0f6a25afab8d63beafab4412349881c8074b4296d not found: ID does not exist" containerID="a6caef4dce9bdad6e10a6bc0f6a25afab8d63beafab4412349881c8074b4296d" Oct 30 20:54:52.025500 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:52.025455 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6caef4dce9bdad6e10a6bc0f6a25afab8d63beafab4412349881c8074b4296d"} err="failed to get container status \"a6caef4dce9bdad6e10a6bc0f6a25afab8d63beafab4412349881c8074b4296d\": rpc error: code = NotFound desc = could not find container \"a6caef4dce9bdad6e10a6bc0f6a25afab8d63beafab4412349881c8074b4296d\": container with ID starting with a6caef4dce9bdad6e10a6bc0f6a25afab8d63beafab4412349881c8074b4296d not found: ID does not exist" Oct 30 20:54:52.031754 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:52.031674 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8"] Oct 30 20:54:52.035547 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:52.035528 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-797b8"] Oct 30 20:54:52.049144 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:52.049098 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj"] Oct 30 20:54:52.051968 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:52.051949 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-mdwqj"] Oct 30 20:54:52.511761 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:54:52.511727 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:54:52.937538 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:52.937494 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3f2d488-7956-4a7a-8b44-53eabadd98bc" path="/var/lib/kubelet/pods/c3f2d488-7956-4a7a-8b44-53eabadd98bc/volumes" Oct 30 20:54:52.938463 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:52.938437 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e029fa76-d526-498e-acbd-efda2b093887" path="/var/lib/kubelet/pods/e029fa76-d526-498e-acbd-efda2b093887/volumes" Oct 30 20:54:55.398155 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.398119 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc"] Oct 30 20:54:55.398649 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.398380 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" podUID="3cd234ce-077b-40b6-ba44-aec166a415aa" containerName="maas-api" containerID="cri-o://8a88e6433b3b97d28ea4fb42b23951469aa9b9e64e59c9014882a4a64eb1835c" gracePeriod=30 Oct 30 20:54:55.552390 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.552351 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk"] Oct 30 20:54:55.552640 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.552592 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" podUID="864fb9de-cedc-4aab-a5ab-c8e038c976fc" containerName="maas-api" containerID="cri-o://748bc469ff35c832ea691673603a755e50f31b4cfb5c0e85ceea7cf6048736c0" gracePeriod=30 Oct 30 20:54:55.721050 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.721022 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" Oct 30 20:54:55.787319 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.787281 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd234ce-077b-40b6-ba44-aec166a415aa-kube-api-access-r7vhc\" (UniqueName: \"kubernetes.io/projected/3cd234ce-077b-40b6-ba44-aec166a415aa-kube-api-access-r7vhc\") pod \"3cd234ce-077b-40b6-ba44-aec166a415aa\" (UID: \"3cd234ce-077b-40b6-ba44-aec166a415aa\") " Oct 30 20:54:55.787496 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.787441 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cd234ce-077b-40b6-ba44-aec166a415aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd234ce-077b-40b6-ba44-aec166a415aa-maas-api-tls\") pod \"3cd234ce-077b-40b6-ba44-aec166a415aa\" (UID: \"3cd234ce-077b-40b6-ba44-aec166a415aa\") " Oct 30 20:54:55.787496 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.787471 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd234ce-077b-40b6-ba44-aec166a415aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd234ce-077b-40b6-ba44-aec166a415aa-openshift-service-ca\") pod \"3cd234ce-077b-40b6-ba44-aec166a415aa\" (UID: \"3cd234ce-077b-40b6-ba44-aec166a415aa\") " Oct 30 20:54:55.787739 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.787523 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cd234ce-077b-40b6-ba44-aec166a415aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd234ce-077b-40b6-ba44-aec166a415aa-metrics-certs\") pod \"3cd234ce-077b-40b6-ba44-aec166a415aa\" (UID: \"3cd234ce-077b-40b6-ba44-aec166a415aa\") " Oct 30 20:54:55.788310 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.788157 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd234ce-077b-40b6-ba44-aec166a415aa-openshift-service-ca" pod "3cd234ce-077b-40b6-ba44-aec166a415aa" (UID: "3cd234ce-077b-40b6-ba44-aec166a415aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:55.788310 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.788281 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd234ce-077b-40b6-ba44-aec166a415aa-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:55.790325 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.790299 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cd234ce-077b-40b6-ba44-aec166a415aa-maas-api-tls" pod "3cd234ce-077b-40b6-ba44-aec166a415aa" (UID: "3cd234ce-077b-40b6-ba44-aec166a415aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:55.791338 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.791313 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cd234ce-077b-40b6-ba44-aec166a415aa-metrics-certs" pod "3cd234ce-077b-40b6-ba44-aec166a415aa" (UID: "3cd234ce-077b-40b6-ba44-aec166a415aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:55.791549 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.791531 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd234ce-077b-40b6-ba44-aec166a415aa-kube-api-access-r7vhc" pod "3cd234ce-077b-40b6-ba44-aec166a415aa" (UID: "3cd234ce-077b-40b6-ba44-aec166a415aa"). InnerVolumeSpecName "kube-api-access-r7vhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:55.836930 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.836901 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" Oct 30 20:54:55.902791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.891445 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/864fb9de-cedc-4aab-a5ab-c8e038c976fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864fb9de-cedc-4aab-a5ab-c8e038c976fc-openshift-service-ca\") pod \"864fb9de-cedc-4aab-a5ab-c8e038c976fc\" (UID: \"864fb9de-cedc-4aab-a5ab-c8e038c976fc\") " Oct 30 20:54:55.902791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.891783 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/864fb9de-cedc-4aab-a5ab-c8e038c976fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864fb9de-cedc-4aab-a5ab-c8e038c976fc-maas-api-tls\") pod \"864fb9de-cedc-4aab-a5ab-c8e038c976fc\" (UID: \"864fb9de-cedc-4aab-a5ab-c8e038c976fc\") " Oct 30 20:54:55.902791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.891931 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/864fb9de-cedc-4aab-a5ab-c8e038c976fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/864fb9de-cedc-4aab-a5ab-c8e038c976fc-metrics-certs\") pod \"864fb9de-cedc-4aab-a5ab-c8e038c976fc\" (UID: \"864fb9de-cedc-4aab-a5ab-c8e038c976fc\") " Oct 30 20:54:55.902791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.891967 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/864fb9de-cedc-4aab-a5ab-c8e038c976fc-kube-api-access-mfqwf\" (UniqueName: \"kubernetes.io/projected/864fb9de-cedc-4aab-a5ab-c8e038c976fc-kube-api-access-mfqwf\") pod \"864fb9de-cedc-4aab-a5ab-c8e038c976fc\" (UID: \"864fb9de-cedc-4aab-a5ab-c8e038c976fc\") " Oct 30 20:54:55.902791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.892557 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cd234ce-077b-40b6-ba44-aec166a415aa-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:55.902791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.892575 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7vhc\" (UniqueName: \"kubernetes.io/projected/3cd234ce-077b-40b6-ba44-aec166a415aa-kube-api-access-r7vhc\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:55.902791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.892591 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd234ce-077b-40b6-ba44-aec166a415aa-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:55.932991 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:55.920983 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864fb9de-cedc-4aab-a5ab-c8e038c976fc-openshift-service-ca" pod "864fb9de-cedc-4aab-a5ab-c8e038c976fc" (UID: "864fb9de-cedc-4aab-a5ab-c8e038c976fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:54:56.085796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.077526 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/864fb9de-cedc-4aab-a5ab-c8e038c976fc-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:56.149761 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.101493 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864fb9de-cedc-4aab-a5ab-c8e038c976fc-maas-api-tls" pod "864fb9de-cedc-4aab-a5ab-c8e038c976fc" (UID: "864fb9de-cedc-4aab-a5ab-c8e038c976fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:56.158867 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.125547 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864fb9de-cedc-4aab-a5ab-c8e038c976fc-kube-api-access-mfqwf" pod "864fb9de-cedc-4aab-a5ab-c8e038c976fc" (UID: "864fb9de-cedc-4aab-a5ab-c8e038c976fc"). InnerVolumeSpecName "kube-api-access-mfqwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:54:56.158867 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.158441 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864fb9de-cedc-4aab-a5ab-c8e038c976fc-metrics-certs" pod "864fb9de-cedc-4aab-a5ab-c8e038c976fc" (UID: "864fb9de-cedc-4aab-a5ab-c8e038c976fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:54:56.161740 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.161709 4848 generic.go:356] "Generic (PLEG): container finished" podID="864fb9de-cedc-4aab-a5ab-c8e038c976fc" containerID="748bc469ff35c832ea691673603a755e50f31b4cfb5c0e85ceea7cf6048736c0" exitCode=0 Oct 30 20:54:56.162342 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.161807 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" Oct 30 20:54:56.162456 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.161829 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" event={"ID":"864fb9de-cedc-4aab-a5ab-c8e038c976fc","Type":"ContainerDied","Data":"748bc469ff35c832ea691673603a755e50f31b4cfb5c0e85ceea7cf6048736c0"} Oct 30 20:54:56.162947 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.162502 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk" event={"ID":"864fb9de-cedc-4aab-a5ab-c8e038c976fc","Type":"ContainerDied","Data":"0d4683383a8a20d51bc2fe8cbffc36fa21c053b232ca4d64d8718847de2b9c40"} Oct 30 20:54:56.162947 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.162544 4848 scope.go:122] "RemoveContainer" containerID="748bc469ff35c832ea691673603a755e50f31b4cfb5c0e85ceea7cf6048736c0" Oct 30 20:54:56.164858 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.164834 4848 generic.go:356] "Generic (PLEG): container finished" podID="3cd234ce-077b-40b6-ba44-aec166a415aa" containerID="8a88e6433b3b97d28ea4fb42b23951469aa9b9e64e59c9014882a4a64eb1835c" exitCode=0 Oct 30 20:54:56.164969 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.164904 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" event={"ID":"3cd234ce-077b-40b6-ba44-aec166a415aa","Type":"ContainerDied","Data":"8a88e6433b3b97d28ea4fb42b23951469aa9b9e64e59c9014882a4a64eb1835c"} Oct 30 20:54:56.164969 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.164941 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" event={"ID":"3cd234ce-077b-40b6-ba44-aec166a415aa","Type":"ContainerDied","Data":"cd3543090a86c7bf34e23d356e1dd359126be835ab4efb83c148a62820bca225"} Oct 30 20:54:56.165302 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.165132 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" Oct 30 20:54:56.176755 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.176731 4848 scope.go:122] "RemoveContainer" containerID="748bc469ff35c832ea691673603a755e50f31b4cfb5c0e85ceea7cf6048736c0" Oct 30 20:54:56.177095 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:54:56.177026 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"748bc469ff35c832ea691673603a755e50f31b4cfb5c0e85ceea7cf6048736c0\": container with ID starting with 748bc469ff35c832ea691673603a755e50f31b4cfb5c0e85ceea7cf6048736c0 not found: ID does not exist" containerID="748bc469ff35c832ea691673603a755e50f31b4cfb5c0e85ceea7cf6048736c0" Oct 30 20:54:56.177095 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.177062 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"748bc469ff35c832ea691673603a755e50f31b4cfb5c0e85ceea7cf6048736c0"} err="failed to get container status \"748bc469ff35c832ea691673603a755e50f31b4cfb5c0e85ceea7cf6048736c0\": rpc error: code = NotFound desc = could not find container \"748bc469ff35c832ea691673603a755e50f31b4cfb5c0e85ceea7cf6048736c0\": container with ID starting with 748bc469ff35c832ea691673603a755e50f31b4cfb5c0e85ceea7cf6048736c0 not found: ID does not exist" Oct 30 20:54:56.177095 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.177085 4848 scope.go:122] "RemoveContainer" containerID="8a88e6433b3b97d28ea4fb42b23951469aa9b9e64e59c9014882a4a64eb1835c" Oct 30 20:54:56.192341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.192281 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk"] Oct 30 20:54:56.194265 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.194247 4848 scope.go:122] "RemoveContainer" containerID="8a88e6433b3b97d28ea4fb42b23951469aa9b9e64e59c9014882a4a64eb1835c" Oct 30 20:54:56.195000 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:54:56.194979 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a88e6433b3b97d28ea4fb42b23951469aa9b9e64e59c9014882a4a64eb1835c\": container with ID starting with 8a88e6433b3b97d28ea4fb42b23951469aa9b9e64e59c9014882a4a64eb1835c not found: ID does not exist" containerID="8a88e6433b3b97d28ea4fb42b23951469aa9b9e64e59c9014882a4a64eb1835c" Oct 30 20:54:56.195085 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.195009 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a88e6433b3b97d28ea4fb42b23951469aa9b9e64e59c9014882a4a64eb1835c"} err="failed to get container status \"8a88e6433b3b97d28ea4fb42b23951469aa9b9e64e59c9014882a4a64eb1835c\": rpc error: code = NotFound desc = could not find container \"8a88e6433b3b97d28ea4fb42b23951469aa9b9e64e59c9014882a4a64eb1835c\": container with ID starting with 8a88e6433b3b97d28ea4fb42b23951469aa9b9e64e59c9014882a4a64eb1835c not found: ID does not exist" Oct 30 20:54:56.196348 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.196322 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-84f901-67d6fd965d-s6qlk"] Oct 30 20:54:56.220848 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.220816 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc"] Oct 30 20:54:56.223807 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.223784 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc"] Oct 30 20:54:56.251668 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.251639 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/864fb9de-cedc-4aab-a5ab-c8e038c976fc-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:56.251837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.251672 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfqwf\" (UniqueName: \"kubernetes.io/projected/864fb9de-cedc-4aab-a5ab-c8e038c976fc-kube-api-access-mfqwf\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:56.251837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.251689 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/864fb9de-cedc-4aab-a5ab-c8e038c976fc-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:54:56.932924 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.932892 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cd234ce-077b-40b6-ba44-aec166a415aa" path="/var/lib/kubelet/pods/3cd234ce-077b-40b6-ba44-aec166a415aa/volumes" Oct 30 20:54:56.933544 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:56.933524 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="864fb9de-cedc-4aab-a5ab-c8e038c976fc" path="/var/lib/kubelet/pods/864fb9de-cedc-4aab-a5ab-c8e038c976fc/volumes" Oct 30 20:54:58.658491 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:54:58.658451 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0333a9-5bd8484697-g9bzc" podUID="3cd234ce-077b-40b6-ba44-aec166a415aa" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.100:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:55:00.955550 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.955517 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d"] Oct 30 20:55:00.956091 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956077 4848 cpu_manager.go:414] "containerMap: removing container" podUID="a79a6860-1766-4d6b-9bb7-ccfa69a02989" containerName="istio-proxy" Oct 30 20:55:00.956136 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956092 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a79a6860-1766-4d6b-9bb7-ccfa69a02989" containerName="istio-proxy" Oct 30 20:55:00.956136 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956110 4848 cpu_manager.go:414] "containerMap: removing container" podUID="e78c510b-f4e1-4d65-93cc-2b00f0896b81" containerName="maas-api" Oct 30 20:55:00.956136 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956115 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78c510b-f4e1-4d65-93cc-2b00f0896b81" containerName="maas-api" Oct 30 20:55:00.956136 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956133 4848 cpu_manager.go:414] "containerMap: removing container" podUID="e029fa76-d526-498e-acbd-efda2b093887" containerName="payload-processing" Oct 30 20:55:00.956256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956138 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e029fa76-d526-498e-acbd-efda2b093887" containerName="payload-processing" Oct 30 20:55:00.956256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956144 4848 cpu_manager.go:414] "containerMap: removing container" podUID="864fb9de-cedc-4aab-a5ab-c8e038c976fc" containerName="maas-api" Oct 30 20:55:00.956256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956148 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="864fb9de-cedc-4aab-a5ab-c8e038c976fc" containerName="maas-api" Oct 30 20:55:00.956256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956157 4848 cpu_manager.go:414] "containerMap: removing container" podUID="3199622d-dbd3-4e39-8376-93322d66b0ae" containerName="istio-proxy" Oct 30 20:55:00.956256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956162 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3199622d-dbd3-4e39-8376-93322d66b0ae" containerName="istio-proxy" Oct 30 20:55:00.956256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956167 4848 cpu_manager.go:414] "containerMap: removing container" podUID="3cd234ce-077b-40b6-ba44-aec166a415aa" containerName="maas-api" Oct 30 20:55:00.956256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956172 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd234ce-077b-40b6-ba44-aec166a415aa" containerName="maas-api" Oct 30 20:55:00.956256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956182 4848 cpu_manager.go:414] "containerMap: removing container" podUID="05c28ea7-d56c-47d6-9a10-531506720261" containerName="payload-processing" Oct 30 20:55:00.956256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956186 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c28ea7-d56c-47d6-9a10-531506720261" containerName="payload-processing" Oct 30 20:55:00.956256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956196 4848 cpu_manager.go:414] "containerMap: removing container" podUID="2c6e17b1-e385-470d-89f0-25397789a650" containerName="payload-processing" Oct 30 20:55:00.956256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956200 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c6e17b1-e385-470d-89f0-25397789a650" containerName="payload-processing" Oct 30 20:55:00.956256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956210 4848 cpu_manager.go:414] "containerMap: removing container" podUID="f616d8f8-b034-4199-8bb0-fe3999e85785" containerName="payload-pre-processing" Oct 30 20:55:00.956256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956214 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f616d8f8-b034-4199-8bb0-fe3999e85785" containerName="payload-pre-processing" Oct 30 20:55:00.956256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956225 4848 cpu_manager.go:414] "containerMap: removing container" podUID="338bef5a-21a8-4a89-8c84-539bdb24482b" containerName="maas-api" Oct 30 20:55:00.956256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956229 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338bef5a-21a8-4a89-8c84-539bdb24482b" containerName="maas-api" Oct 30 20:55:00.956256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956237 4848 cpu_manager.go:414] "containerMap: removing container" podUID="bbbf9456-036c-4acb-9d73-662428928fa8" containerName="payload-pre-processing" Oct 30 20:55:00.956256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956241 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbf9456-036c-4acb-9d73-662428928fa8" containerName="payload-pre-processing" Oct 30 20:55:00.956256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956251 4848 cpu_manager.go:414] "containerMap: removing container" podUID="c3f2d488-7956-4a7a-8b44-53eabadd98bc" containerName="payload-pre-processing" Oct 30 20:55:00.956256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956256 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3f2d488-7956-4a7a-8b44-53eabadd98bc" containerName="payload-pre-processing" Oct 30 20:55:00.956256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956264 4848 cpu_manager.go:414] "containerMap: removing container" podUID="6c39353d-db8a-4650-bf95-3ca9bc3be519" containerName="maas-api" Oct 30 20:55:00.956863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956268 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c39353d-db8a-4650-bf95-3ca9bc3be519" containerName="maas-api" Oct 30 20:55:00.956863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956276 4848 cpu_manager.go:414] "containerMap: removing container" podUID="706abb91-d2cd-4f79-8247-2f9077a602c6" containerName="maas-api" Oct 30 20:55:00.956863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956280 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="706abb91-d2cd-4f79-8247-2f9077a602c6" containerName="maas-api" Oct 30 20:55:00.956863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956369 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbf9456-036c-4acb-9d73-662428928fa8" containerName="payload-pre-processing" Oct 30 20:55:00.956863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956380 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="3199622d-dbd3-4e39-8376-93322d66b0ae" containerName="istio-proxy" Oct 30 20:55:00.956863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956385 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3f2d488-7956-4a7a-8b44-53eabadd98bc" containerName="payload-pre-processing" Oct 30 20:55:00.956863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956392 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c39353d-db8a-4650-bf95-3ca9bc3be519" containerName="maas-api" Oct 30 20:55:00.956863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956398 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="706abb91-d2cd-4f79-8247-2f9077a602c6" containerName="maas-api" Oct 30 20:55:00.956863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956403 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="a79a6860-1766-4d6b-9bb7-ccfa69a02989" containerName="istio-proxy" Oct 30 20:55:00.956863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956408 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="a79a6860-1766-4d6b-9bb7-ccfa69a02989" containerName="istio-proxy" Oct 30 20:55:00.956863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956414 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78c510b-f4e1-4d65-93cc-2b00f0896b81" containerName="maas-api" Oct 30 20:55:00.956863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956421 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="e029fa76-d526-498e-acbd-efda2b093887" containerName="payload-processing" Oct 30 20:55:00.956863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956426 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="864fb9de-cedc-4aab-a5ab-c8e038c976fc" containerName="maas-api" Oct 30 20:55:00.956863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956432 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd234ce-077b-40b6-ba44-aec166a415aa" containerName="maas-api" Oct 30 20:55:00.956863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956437 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c28ea7-d56c-47d6-9a10-531506720261" containerName="payload-processing" Oct 30 20:55:00.956863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956444 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c6e17b1-e385-470d-89f0-25397789a650" containerName="payload-processing" Oct 30 20:55:00.956863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956449 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="f616d8f8-b034-4199-8bb0-fe3999e85785" containerName="payload-pre-processing" Oct 30 20:55:00.956863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.956456 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="338bef5a-21a8-4a89-8c84-539bdb24482b" containerName="maas-api" Oct 30 20:55:00.961796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.961753 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d" Oct 30 20:55:00.967745 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.967721 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d"] Oct 30 20:55:00.968381 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.968353 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a52bd2a1\"" Oct 30 20:55:00.968505 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:00.968412 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a52bd2a1-dockercfg-v77n6\"" Oct 30 20:55:01.007579 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.007542 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g"] Oct 30 20:55:01.008181 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.008162 4848 cpu_manager.go:414] "containerMap: removing container" podUID="a79a6860-1766-4d6b-9bb7-ccfa69a02989" containerName="istio-proxy" Oct 30 20:55:01.008181 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.008178 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a79a6860-1766-4d6b-9bb7-ccfa69a02989" containerName="istio-proxy" Oct 30 20:55:01.008351 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.008205 4848 cpu_manager.go:414] "containerMap: removing container" podUID="3199622d-dbd3-4e39-8376-93322d66b0ae" containerName="istio-proxy" Oct 30 20:55:01.008351 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.008211 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3199622d-dbd3-4e39-8376-93322d66b0ae" containerName="istio-proxy" Oct 30 20:55:01.008351 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.008313 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="3199622d-dbd3-4e39-8376-93322d66b0ae" containerName="istio-proxy" Oct 30 20:55:01.011947 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.011925 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g" Oct 30 20:55:01.020734 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.020706 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g"] Oct 30 20:55:01.036383 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.036354 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6069037-4447-4cda-be07-3323e5cdf074-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d\" (UID: \"f6069037-4447-4cda-be07-3323e5cdf074\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d" Oct 30 20:55:01.036511 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.036458 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rljpj\" (UniqueName: \"kubernetes.io/projected/f6069037-4447-4cda-be07-3323e5cdf074-kube-api-access-rljpj\") pod \"payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d\" (UID: \"f6069037-4447-4cda-be07-3323e5cdf074\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d" Oct 30 20:55:01.150788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.137691 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6069037-4447-4cda-be07-3323e5cdf074-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d\" (UID: \"f6069037-4447-4cda-be07-3323e5cdf074\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d" Oct 30 20:55:01.150788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.137815 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rljpj\" (UniqueName: \"kubernetes.io/projected/f6069037-4447-4cda-be07-3323e5cdf074-kube-api-access-rljpj\") pod \"payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d\" (UID: \"f6069037-4447-4cda-be07-3323e5cdf074\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d" Oct 30 20:55:01.150788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.139194 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6069037-4447-4cda-be07-3323e5cdf074-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d\" (UID: \"f6069037-4447-4cda-be07-3323e5cdf074\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d" Oct 30 20:55:01.180054 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.177100 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rljpj\" (UniqueName: \"kubernetes.io/projected/f6069037-4447-4cda-be07-3323e5cdf074-kube-api-access-rljpj\") pod \"payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d\" (UID: \"f6069037-4447-4cda-be07-3323e5cdf074\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d" Oct 30 20:55:01.240498 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.240404 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/228304aa-a745-45d1-a305-d252cf89aff3-plugins-config-volume\") pod \"payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g\" (UID: \"228304aa-a745-45d1-a305-d252cf89aff3\") " pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g" Oct 30 20:55:01.245833 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.240836 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rdzt\" (UniqueName: \"kubernetes.io/projected/228304aa-a745-45d1-a305-d252cf89aff3-kube-api-access-6rdzt\") pod \"payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g\" (UID: \"228304aa-a745-45d1-a305-d252cf89aff3\") " pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g" Oct 30 20:55:01.280828 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.280796 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d" Oct 30 20:55:01.348931 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.348885 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rdzt\" (UniqueName: \"kubernetes.io/projected/228304aa-a745-45d1-a305-d252cf89aff3-kube-api-access-6rdzt\") pod \"payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g\" (UID: \"228304aa-a745-45d1-a305-d252cf89aff3\") " pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g" Oct 30 20:55:01.349525 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.349501 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/228304aa-a745-45d1-a305-d252cf89aff3-plugins-config-volume\") pod \"payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g\" (UID: \"228304aa-a745-45d1-a305-d252cf89aff3\") " pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g" Oct 30 20:55:01.350549 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.350528 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/228304aa-a745-45d1-a305-d252cf89aff3-plugins-config-volume\") pod \"payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g\" (UID: \"228304aa-a745-45d1-a305-d252cf89aff3\") " pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g" Oct 30 20:55:01.366285 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.366254 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rdzt\" (UniqueName: \"kubernetes.io/projected/228304aa-a745-45d1-a305-d252cf89aff3-kube-api-access-6rdzt\") pod \"payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g\" (UID: \"228304aa-a745-45d1-a305-d252cf89aff3\") " pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g" Oct 30 20:55:01.629526 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.629427 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g" Oct 30 20:55:01.642229 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.642202 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d"] Oct 30 20:55:01.644411 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:55:01.644380 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6069037_4447_4cda_be07_3323e5cdf074.slice/crio-b197d0865b9413afc1873ac1878c4d7775cfd215c48de83120c9adaa2dbe2551 WatchSource:0}: Error finding container b197d0865b9413afc1873ac1878c4d7775cfd215c48de83120c9adaa2dbe2551: Status 404 returned error can't find the container with id b197d0865b9413afc1873ac1878c4d7775cfd215c48de83120c9adaa2dbe2551 Oct 30 20:55:01.990421 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:01.990230 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g"] Oct 30 20:55:02.255412 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:02.255314 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g" event={"ID":"228304aa-a745-45d1-a305-d252cf89aff3","Type":"ContainerStarted","Data":"7c4d4f823036297b997b69f37e75ee11697dba45aa4010d2c532373e86c3f10e"} Oct 30 20:55:02.255412 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:02.255382 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g" event={"ID":"228304aa-a745-45d1-a305-d252cf89aff3","Type":"ContainerStarted","Data":"1edb6bcac502d35abacb14e601856a4f7f653c64b7d03eb148fc47aeacaa246c"} Oct 30 20:55:02.255645 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:02.255540 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g" Oct 30 20:55:02.257565 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:02.257541 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d" event={"ID":"f6069037-4447-4cda-be07-3323e5cdf074","Type":"ContainerStarted","Data":"28642b2258c5c25c9c0701b44f680331a9775eddd7564db7a2435e39ff4dcd95"} Oct 30 20:55:02.257674 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:02.257583 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d" event={"ID":"f6069037-4447-4cda-be07-3323e5cdf074","Type":"ContainerStarted","Data":"b197d0865b9413afc1873ac1878c4d7775cfd215c48de83120c9adaa2dbe2551"} Oct 30 20:55:02.257674 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:02.257610 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d" Oct 30 20:55:02.274466 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:02.274424 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g" podStartSLOduration=2.274413323 podStartE2EDuration="2.274413323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:02.270697747 +0000 UTC m=+1007.450594253" watchObservedRunningTime="2026-10-30 20:55:02.274413323 +0000 UTC m=+1007.454309829" Oct 30 20:55:02.288689 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:02.288646 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d" podStartSLOduration=2.288635036 podStartE2EDuration="2.288635036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:02.285980587 +0000 UTC m=+1007.465877094" watchObservedRunningTime="2026-10-30 20:55:02.288635036 +0000 UTC m=+1007.468531543" Oct 30 20:55:04.040957 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:04.040921 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5"] Oct 30 20:55:04.041364 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:04.041124 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" podUID="d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" containerName="istio-proxy" containerID="cri-o://06380184dc77630cbccc9555f3684de060f50eafa66b0efe759d2b7a95d5b991" gracePeriod=30 Oct 30 20:55:04.318841 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:04.318620 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c"] Oct 30 20:55:04.319017 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:04.318883 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" podUID="feb15caf-387e-40ab-a8f8-d6292cf0b28b" containerName="istio-proxy" containerID="cri-o://3692cbefcc56c0cf2db24005b033c48a2c36896790c1a37375ec4e0d2770bae4" gracePeriod=30 Oct 30 20:55:06.194655 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:06.194625 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4"] Oct 30 20:55:06.199972 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:06.199953 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" Oct 30 20:55:06.203287 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:06.203266 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a52bd2a1-serving-cert\"" Oct 30 20:55:06.206264 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:06.206230 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4"] Oct 30 20:55:06.267026 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:06.266998 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-maas-api-tls\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4\" (UID: \"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" Oct 30 20:55:06.267026 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:06.267029 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-openshift-service-ca\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4\" (UID: \"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" Oct 30 20:55:06.267228 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:06.267069 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-metrics-certs\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4\" (UID: \"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" Oct 30 20:55:06.267270 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:06.267241 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f9f8\" (UniqueName: \"kubernetes.io/projected/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-kube-api-access-5f9f8\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4\" (UID: \"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" Oct 30 20:55:06.370962 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:06.370904 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-metrics-certs\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4\" (UID: \"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" Oct 30 20:55:06.391464 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:06.391426 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f9f8\" (UniqueName: \"kubernetes.io/projected/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-kube-api-access-5f9f8\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4\" (UID: \"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" Oct 30 20:55:06.392828 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:06.392795 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-maas-api-tls\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4\" (UID: \"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" Oct 30 20:55:06.393002 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:06.392985 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-openshift-service-ca\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4\" (UID: \"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" Oct 30 20:55:06.394148 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:06.394125 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-openshift-service-ca\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4\" (UID: \"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" Oct 30 20:55:06.412509 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:06.412479 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f9f8\" (UniqueName: \"kubernetes.io/projected/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-kube-api-access-5f9f8\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4\" (UID: \"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" Oct 30 20:55:06.418808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:06.414428 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-metrics-certs\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4\" (UID: \"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" Oct 30 20:55:06.419577 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:06.419545 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-maas-api-tls\") pod \"maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4\" (UID: \"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" Oct 30 20:55:06.519436 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:06.519354 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" Oct 30 20:55:06.801911 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:06.801882 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4"] Oct 30 20:55:06.803272 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:55:06.803244 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1e0e3a_b4f2_4537_848e_fdf5450dc36a.slice/crio-3543da9085d884c2004622cdbe4c8008a46d081f1b4fa336f48862612671cf29 WatchSource:0}: Error finding container 3543da9085d884c2004622cdbe4c8008a46d081f1b4fa336f48862612671cf29: Status 404 returned error can't find the container with id 3543da9085d884c2004622cdbe4c8008a46d081f1b4fa336f48862612671cf29 Oct 30 20:55:07.440107 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.440073 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" event={"ID":"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a","Type":"ContainerStarted","Data":"3543da9085d884c2004622cdbe4c8008a46d081f1b4fa336f48862612671cf29"} Oct 30 20:55:07.462184 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.462154 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw"] Oct 30 20:55:07.469399 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.469284 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.490832 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.487653 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5e71c9-openshift-default-dockercfg-hlrj6\"" Oct 30 20:55:07.492938 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.492911 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw"] Oct 30 20:55:07.523754 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.523513 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-podinfo\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.523754 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.523589 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9525d3-a53f-4e30-8fae-12a407eace4e-istiod-ca-cert\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.523754 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.523711 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-data\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.523754 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.523751 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-token\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.524049 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.523895 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-envoy\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.524049 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.523960 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl7x4\" (UniqueName: \"kubernetes.io/projected/bf9525d3-a53f-4e30-8fae-12a407eace4e-kube-api-access-jl7x4\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.524049 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.524005 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-credential-socket\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.524049 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.524034 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-workload-socket\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.524220 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.524060 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-workload-certs\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.577214 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.577179 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8"] Oct 30 20:55:07.581948 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.581925 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.586366 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.586342 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-eebcb4-openshift-default-dockercfg-nsq5z\"" Oct 30 20:55:07.595400 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.594551 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8"] Oct 30 20:55:07.625630 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.625600 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-token\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.625825 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.625713 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-istio-data\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.625955 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.625934 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-envoy\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.626447 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.626031 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl7x4\" (UniqueName: \"kubernetes.io/projected/bf9525d3-a53f-4e30-8fae-12a407eace4e-kube-api-access-jl7x4\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.626447 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.626096 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-workload-certs\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.626447 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.626142 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-credential-socket\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.626447 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.626177 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-workload-socket\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.626447 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.626205 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-workload-certs\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.627055 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.627031 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/548c8714-1683-4286-865d-96d9d76cd5ed-istiod-ca-cert\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.627408 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.627388 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-credential-socket\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.627516 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.627498 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-istio-envoy\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.627615 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.627592 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-workload-certs\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.627672 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.627633 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-credential-socket\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.627747 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.627724 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kjhg\" (UniqueName: \"kubernetes.io/projected/548c8714-1683-4286-865d-96d9d76cd5ed-kube-api-access-5kjhg\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.627850 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.627814 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-podinfo\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.627907 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.627853 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/548c8714-1683-4286-865d-96d9d76cd5ed-istio-token\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.627907 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.627864 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-workload-socket\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.627907 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.627882 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-workload-socket\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.628067 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.627907 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/548c8714-1683-4286-865d-96d9d76cd5ed-istio-podinfo\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.628184 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.628162 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9525d3-a53f-4e30-8fae-12a407eace4e-istiod-ca-cert\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.628261 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.628240 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-data\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.628800 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.628745 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-data\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.628969 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.628871 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9525d3-a53f-4e30-8fae-12a407eace4e-istiod-ca-cert\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.633374 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.633279 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-envoy\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.633654 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.633576 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-podinfo\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.638687 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.638641 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-token\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.640230 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.640198 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl7x4\" (UniqueName: \"kubernetes.io/projected/bf9525d3-a53f-4e30-8fae-12a407eace4e-kube-api-access-jl7x4\") pod \"e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.740912 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.734843 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-istio-data\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.741262 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.741237 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-workload-certs\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.752632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.741727 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/548c8714-1683-4286-865d-96d9d76cd5ed-istiod-ca-cert\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.752915 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.752890 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-istio-envoy\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.756788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.753498 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-credential-socket\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.785908 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.785876 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kjhg\" (UniqueName: \"kubernetes.io/projected/548c8714-1683-4286-865d-96d9d76cd5ed-kube-api-access-5kjhg\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.786231 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.786204 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/548c8714-1683-4286-865d-96d9d76cd5ed-istio-token\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.786679 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.786658 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-workload-socket\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.787172 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.787150 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/548c8714-1683-4286-865d-96d9d76cd5ed-istio-podinfo\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.787974 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.786072 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-istio-envoy\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.788155 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.741643 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-workload-certs\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.788252 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.740379 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-istio-data\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.788431 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.773782 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-credential-socket\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.788554 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.787098 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-workload-socket\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.788650 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.775138 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/548c8714-1683-4286-865d-96d9d76cd5ed-istiod-ca-cert\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.819163 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.818910 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:07.823786 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.819869 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/548c8714-1683-4286-865d-96d9d76cd5ed-istio-podinfo\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.862959 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.846404 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/548c8714-1683-4286-865d-96d9d76cd5ed-istio-token\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.862959 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.852728 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kjhg\" (UniqueName: \"kubernetes.io/projected/548c8714-1683-4286-865d-96d9d76cd5ed-kube-api-access-5kjhg\") pod \"e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:07.909386 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:07.909345 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:08.336859 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:08.336833 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw"] Oct 30 20:55:08.338455 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:55:08.338429 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf9525d3_a53f_4e30_8fae_12a407eace4e.slice/crio-d0b3278513c2fc1b03cffd2f6149c4e126abf4f258470692cd0df4caa3a04a27 WatchSource:0}: Error finding container d0b3278513c2fc1b03cffd2f6149c4e126abf4f258470692cd0df4caa3a04a27: Status 404 returned error can't find the container with id d0b3278513c2fc1b03cffd2f6149c4e126abf4f258470692cd0df4caa3a04a27 Oct 30 20:55:08.340553 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:08.340509 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:08.340649 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:08.340598 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:08.340703 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:08.340644 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:08.344976 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:08.344955 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8"] Oct 30 20:55:08.345337 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:55:08.345319 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548c8714_1683_4286_865d_96d9d76cd5ed.slice/crio-7c6dd11495c83d8f85de17ee42730e64cee7e67e0d7cb59e65f3d8ef40261330 WatchSource:0}: Error finding container 7c6dd11495c83d8f85de17ee42730e64cee7e67e0d7cb59e65f3d8ef40261330: Status 404 returned error can't find the container with id 7c6dd11495c83d8f85de17ee42730e64cee7e67e0d7cb59e65f3d8ef40261330 Oct 30 20:55:08.350819 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:08.350792 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:08.350910 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:08.350851 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:08.350910 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:08.350879 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:08.503826 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:08.503787 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" event={"ID":"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a","Type":"ContainerStarted","Data":"eac9c8ddddd077b77463b796add53b9dc3bfeba6551731eaad8286aabe196959"} Oct 30 20:55:08.505799 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:08.505645 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" Oct 30 20:55:08.508896 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:08.508866 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" event={"ID":"548c8714-1683-4286-865d-96d9d76cd5ed","Type":"ContainerStarted","Data":"d965eb356a54058eb8f35428787584c97c196c994ae44c1d0ecb2679ba1d286d"} Oct 30 20:55:08.509104 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:08.509080 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" event={"ID":"548c8714-1683-4286-865d-96d9d76cd5ed","Type":"ContainerStarted","Data":"7c6dd11495c83d8f85de17ee42730e64cee7e67e0d7cb59e65f3d8ef40261330"} Oct 30 20:55:08.516446 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:08.516410 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" event={"ID":"bf9525d3-a53f-4e30-8fae-12a407eace4e","Type":"ContainerStarted","Data":"1fe0afeb0b041b6f6c051e19cebc0b21d1e239f02ac85013d8ede4f6396dfe74"} Oct 30 20:55:08.516703 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:08.516679 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" event={"ID":"bf9525d3-a53f-4e30-8fae-12a407eace4e","Type":"ContainerStarted","Data":"d0b3278513c2fc1b03cffd2f6149c4e126abf4f258470692cd0df4caa3a04a27"} Oct 30 20:55:08.535750 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:08.535692 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" podStartSLOduration=1.902636432 podStartE2EDuration="2.535673563s" totalImagesPullingTime="633.037131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:08.528039895 +0000 UTC m=+1013.707936401" watchObservedRunningTime="2026-10-30 20:55:08.535673563 +0000 UTC m=+1013.715570071" Oct 30 20:55:08.562451 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:08.562389 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" podStartSLOduration=1.56236993 podStartE2EDuration="1.56236993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:08.5583213 +0000 UTC m=+1013.738217809" watchObservedRunningTime="2026-10-30 20:55:08.56236993 +0000 UTC m=+1013.742266439" Oct 30 20:55:08.821714 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:08.821676 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:08.910083 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:08.910053 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:09.316789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.316740 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:55:09.340289 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.340235 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" podStartSLOduration=2.340220246 podStartE2EDuration="2.340220246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:08.584735742 +0000 UTC m=+1013.764632252" watchObservedRunningTime="2026-10-30 20:55:09.340220246 +0000 UTC m=+1014.520116753" Oct 30 20:55:09.378295 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.378263 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-workload-certs\") pod \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " Oct 30 20:55:09.378295 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.378298 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-envoy\") pod \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " Oct 30 20:55:09.378594 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.378357 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-data\") pod \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " Oct 30 20:55:09.378594 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.378430 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-token\" (UniqueName: \"kubernetes.io/projected/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-token\") pod \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " Oct 30 20:55:09.378594 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.378476 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-kube-api-access-msxz7\" (UniqueName: \"kubernetes.io/projected/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-kube-api-access-msxz7\") pod \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " Oct 30 20:55:09.378594 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.378514 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-workload-socket\") pod \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " Oct 30 20:55:09.378594 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.378561 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-credential-socket\") pod \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " Oct 30 20:55:09.378923 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.378644 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-workload-certs" pod "d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" (UID: "d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:09.378923 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.378667 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istiod-ca-cert\") pod \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " Oct 30 20:55:09.378923 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.378753 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-podinfo\") pod \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\" (UID: \"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4\") " Oct 30 20:55:09.379363 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.379336 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-data" pod "d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" (UID: "d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:09.379510 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.379417 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.379572 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.379512 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-workload-socket" pod "d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" (UID: "d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:09.379782 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.379747 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-credential-socket" pod "d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" (UID: "d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:09.379942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.379916 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istiod-ca-cert" pod "d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" (UID: "d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:09.380814 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.380791 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-envoy" pod "d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" (UID: "d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:09.381271 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.381249 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-podinfo" pod "d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" (UID: "d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:55:09.381744 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.381718 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-token" pod "d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" (UID: "d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:09.381918 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.381901 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-kube-api-access-msxz7" pod "d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" (UID: "d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4"). InnerVolumeSpecName "kube-api-access-msxz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:09.482331 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.482179 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.482331 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.482220 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.482331 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.482235 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.482331 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.482248 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.482331 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.482261 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.482331 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.482274 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.482331 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.482288 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msxz7\" (UniqueName: \"kubernetes.io/projected/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-kube-api-access-msxz7\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.482331 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.482303 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.558814 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.555102 4848 generic.go:356] "Generic (PLEG): container finished" podID="d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" containerID="06380184dc77630cbccc9555f3684de060f50eafa66b0efe759d2b7a95d5b991" exitCode=0 Oct 30 20:55:09.558814 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.556159 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" Oct 30 20:55:09.561925 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.560485 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" event={"ID":"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4","Type":"ContainerDied","Data":"06380184dc77630cbccc9555f3684de060f50eafa66b0efe759d2b7a95d5b991"} Oct 30 20:55:09.561925 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.560542 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5" event={"ID":"d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4","Type":"ContainerDied","Data":"8758eb869b723144beab15c4b2c05a7e3b636c47dfb282e1558d02d7b58e9c86"} Oct 30 20:55:09.561925 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.560572 4848 scope.go:122] "RemoveContainer" containerID="06380184dc77630cbccc9555f3684de060f50eafa66b0efe759d2b7a95d5b991" Oct 30 20:55:09.576961 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.576936 4848 scope.go:122] "RemoveContainer" containerID="250b55bf2b87b5be4f405ed58af04e3ecc551408a3c037b91fd77c4b3d92bd2e" Oct 30 20:55:09.592067 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.591932 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5"] Oct 30 20:55:09.592381 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.592361 4848 scope.go:122] "RemoveContainer" containerID="06380184dc77630cbccc9555f3684de060f50eafa66b0efe759d2b7a95d5b991" Oct 30 20:55:09.592719 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:55:09.592698 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06380184dc77630cbccc9555f3684de060f50eafa66b0efe759d2b7a95d5b991\": container with ID starting with 06380184dc77630cbccc9555f3684de060f50eafa66b0efe759d2b7a95d5b991 not found: ID does not exist" containerID="06380184dc77630cbccc9555f3684de060f50eafa66b0efe759d2b7a95d5b991" Oct 30 20:55:09.592843 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.592724 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06380184dc77630cbccc9555f3684de060f50eafa66b0efe759d2b7a95d5b991"} err="failed to get container status \"06380184dc77630cbccc9555f3684de060f50eafa66b0efe759d2b7a95d5b991\": rpc error: code = NotFound desc = could not find container \"06380184dc77630cbccc9555f3684de060f50eafa66b0efe759d2b7a95d5b991\": container with ID starting with 06380184dc77630cbccc9555f3684de060f50eafa66b0efe759d2b7a95d5b991 not found: ID does not exist" Oct 30 20:55:09.592843 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.592786 4848 scope.go:122] "RemoveContainer" containerID="250b55bf2b87b5be4f405ed58af04e3ecc551408a3c037b91fd77c4b3d92bd2e" Oct 30 20:55:09.593070 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:55:09.593048 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"250b55bf2b87b5be4f405ed58af04e3ecc551408a3c037b91fd77c4b3d92bd2e\": container with ID starting with 250b55bf2b87b5be4f405ed58af04e3ecc551408a3c037b91fd77c4b3d92bd2e not found: ID does not exist" containerID="250b55bf2b87b5be4f405ed58af04e3ecc551408a3c037b91fd77c4b3d92bd2e" Oct 30 20:55:09.593141 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.593072 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"250b55bf2b87b5be4f405ed58af04e3ecc551408a3c037b91fd77c4b3d92bd2e"} err="failed to get container status \"250b55bf2b87b5be4f405ed58af04e3ecc551408a3c037b91fd77c4b3d92bd2e\": rpc error: code = NotFound desc = could not find container \"250b55bf2b87b5be4f405ed58af04e3ecc551408a3c037b91fd77c4b3d92bd2e\": container with ID starting with 250b55bf2b87b5be4f405ed58af04e3ecc551408a3c037b91fd77c4b3d92bd2e not found: ID does not exist" Oct 30 20:55:09.595413 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.595389 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0333a9-openshift-default-54cffbc65-q7hx5"] Oct 30 20:55:09.739461 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.739436 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:55:09.786484 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.786453 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-workload-socket\") pod \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " Oct 30 20:55:09.786644 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.786506 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-credential-socket\") pod \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " Oct 30 20:55:09.786644 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.786579 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-envoy\") pod \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " Oct 30 20:55:09.786644 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.786617 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-workload-certs\") pod \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " Oct 30 20:55:09.786853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.786670 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istiod-ca-cert\") pod \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " Oct 30 20:55:09.786853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.786752 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feb15caf-387e-40ab-a8f8-d6292cf0b28b-kube-api-access-nvqzz\" (UniqueName: \"kubernetes.io/projected/feb15caf-387e-40ab-a8f8-d6292cf0b28b-kube-api-access-nvqzz\") pod \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " Oct 30 20:55:09.786853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.786807 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-workload-socket" pod "feb15caf-387e-40ab-a8f8-d6292cf0b28b" (UID: "feb15caf-387e-40ab-a8f8-d6292cf0b28b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:09.786853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.786822 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-data\") pod \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " Oct 30 20:55:09.787049 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.786863 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-podinfo\") pod \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " Oct 30 20:55:09.787049 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.786988 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-token\" (UniqueName: \"kubernetes.io/projected/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-token\") pod \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\" (UID: \"feb15caf-387e-40ab-a8f8-d6292cf0b28b\") " Oct 30 20:55:09.787186 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.787160 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istiod-ca-cert" pod "feb15caf-387e-40ab-a8f8-d6292cf0b28b" (UID: "feb15caf-387e-40ab-a8f8-d6292cf0b28b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:09.787424 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.787334 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-credential-socket" pod "feb15caf-387e-40ab-a8f8-d6292cf0b28b" (UID: "feb15caf-387e-40ab-a8f8-d6292cf0b28b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:09.787537 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.787514 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-workload-certs" pod "feb15caf-387e-40ab-a8f8-d6292cf0b28b" (UID: "feb15caf-387e-40ab-a8f8-d6292cf0b28b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:09.787615 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.787565 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-data" pod "feb15caf-387e-40ab-a8f8-d6292cf0b28b" (UID: "feb15caf-387e-40ab-a8f8-d6292cf0b28b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:09.787980 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.787959 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.788096 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.788082 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.788185 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.788174 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.788268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.788257 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.788375 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.788362 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.789881 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.789853 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-envoy" pod "feb15caf-387e-40ab-a8f8-d6292cf0b28b" (UID: "feb15caf-387e-40ab-a8f8-d6292cf0b28b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:55:09.790066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.790041 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-podinfo" pod "feb15caf-387e-40ab-a8f8-d6292cf0b28b" (UID: "feb15caf-387e-40ab-a8f8-d6292cf0b28b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:55:09.790735 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.790700 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb15caf-387e-40ab-a8f8-d6292cf0b28b-kube-api-access-nvqzz" pod "feb15caf-387e-40ab-a8f8-d6292cf0b28b" (UID: "feb15caf-387e-40ab-a8f8-d6292cf0b28b"). InnerVolumeSpecName "kube-api-access-nvqzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:09.791480 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.791460 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-token" pod "feb15caf-387e-40ab-a8f8-d6292cf0b28b" (UID: "feb15caf-387e-40ab-a8f8-d6292cf0b28b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:09.827419 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.827386 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:09.889986 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.889958 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.889986 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.889983 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvqzz\" (UniqueName: \"kubernetes.io/projected/feb15caf-387e-40ab-a8f8-d6292cf0b28b-kube-api-access-nvqzz\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.890178 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.889995 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.890178 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.890004 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feb15caf-387e-40ab-a8f8-d6292cf0b28b-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:09.916251 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:09.916222 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:10.566532 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:10.566501 4848 generic.go:356] "Generic (PLEG): container finished" podID="feb15caf-387e-40ab-a8f8-d6292cf0b28b" containerID="3692cbefcc56c0cf2db24005b033c48a2c36896790c1a37375ec4e0d2770bae4" exitCode=0 Oct 30 20:55:10.566985 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:10.566548 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" Oct 30 20:55:10.566985 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:10.566588 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" event={"ID":"feb15caf-387e-40ab-a8f8-d6292cf0b28b","Type":"ContainerDied","Data":"3692cbefcc56c0cf2db24005b033c48a2c36896790c1a37375ec4e0d2770bae4"} Oct 30 20:55:10.566985 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:10.566645 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c" event={"ID":"feb15caf-387e-40ab-a8f8-d6292cf0b28b","Type":"ContainerDied","Data":"b40a8371b5d26b683e09d4ff9daaa4eece225d77bf3d2522f47887fcd09a3385"} Oct 30 20:55:10.566985 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:10.566670 4848 scope.go:122] "RemoveContainer" containerID="3692cbefcc56c0cf2db24005b033c48a2c36896790c1a37375ec4e0d2770bae4" Oct 30 20:55:10.567305 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:10.567285 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:10.567570 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:10.567549 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:10.568264 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:10.568241 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:10.568606 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:10.568587 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:10.577715 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:10.577701 4848 scope.go:122] "RemoveContainer" containerID="93404305a665ba336762add05864fe32554a5c83036b3c9ba6a1e3c6121b29e4" Oct 30 20:55:10.586895 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:10.586881 4848 scope.go:122] "RemoveContainer" containerID="3692cbefcc56c0cf2db24005b033c48a2c36896790c1a37375ec4e0d2770bae4" Oct 30 20:55:10.587150 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:55:10.587127 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3692cbefcc56c0cf2db24005b033c48a2c36896790c1a37375ec4e0d2770bae4\": container with ID starting with 3692cbefcc56c0cf2db24005b033c48a2c36896790c1a37375ec4e0d2770bae4 not found: ID does not exist" containerID="3692cbefcc56c0cf2db24005b033c48a2c36896790c1a37375ec4e0d2770bae4" Oct 30 20:55:10.587195 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:10.587149 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3692cbefcc56c0cf2db24005b033c48a2c36896790c1a37375ec4e0d2770bae4"} err="failed to get container status \"3692cbefcc56c0cf2db24005b033c48a2c36896790c1a37375ec4e0d2770bae4\": rpc error: code = NotFound desc = could not find container \"3692cbefcc56c0cf2db24005b033c48a2c36896790c1a37375ec4e0d2770bae4\": container with ID starting with 3692cbefcc56c0cf2db24005b033c48a2c36896790c1a37375ec4e0d2770bae4 not found: ID does not exist" Oct 30 20:55:10.587195 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:10.587164 4848 scope.go:122] "RemoveContainer" containerID="93404305a665ba336762add05864fe32554a5c83036b3c9ba6a1e3c6121b29e4" Oct 30 20:55:10.587396 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:55:10.587377 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93404305a665ba336762add05864fe32554a5c83036b3c9ba6a1e3c6121b29e4\": container with ID starting with 93404305a665ba336762add05864fe32554a5c83036b3c9ba6a1e3c6121b29e4 not found: ID does not exist" containerID="93404305a665ba336762add05864fe32554a5c83036b3c9ba6a1e3c6121b29e4" Oct 30 20:55:10.587441 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:10.587405 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93404305a665ba336762add05864fe32554a5c83036b3c9ba6a1e3c6121b29e4"} err="failed to get container status \"93404305a665ba336762add05864fe32554a5c83036b3c9ba6a1e3c6121b29e4\": rpc error: code = NotFound desc = could not find container \"93404305a665ba336762add05864fe32554a5c83036b3c9ba6a1e3c6121b29e4\": container with ID starting with 93404305a665ba336762add05864fe32554a5c83036b3c9ba6a1e3c6121b29e4 not found: ID does not exist" Oct 30 20:55:10.632339 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:10.632271 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c"] Oct 30 20:55:10.642682 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:10.642646 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-84f901-openshift-default-58866446d7-xfr7c"] Oct 30 20:55:10.933009 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:10.932977 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" path="/var/lib/kubelet/pods/d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4/volumes" Oct 30 20:55:10.933515 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:10.933501 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feb15caf-387e-40ab-a8f8-d6292cf0b28b" path="/var/lib/kubelet/pods/feb15caf-387e-40ab-a8f8-d6292cf0b28b/volumes" Oct 30 20:55:12.070232 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.070198 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m"] Oct 30 20:55:12.070866 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.070852 4848 cpu_manager.go:414] "containerMap: removing container" podUID="feb15caf-387e-40ab-a8f8-d6292cf0b28b" containerName="istio-proxy" Oct 30 20:55:12.070915 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.070867 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feb15caf-387e-40ab-a8f8-d6292cf0b28b" containerName="istio-proxy" Oct 30 20:55:12.070915 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.070901 4848 cpu_manager.go:414] "containerMap: removing container" podUID="d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" containerName="istio-proxy" Oct 30 20:55:12.070915 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.070906 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" containerName="istio-proxy" Oct 30 20:55:12.071019 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.070918 4848 cpu_manager.go:414] "containerMap: removing container" podUID="d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" containerName="istio-proxy" Oct 30 20:55:12.071019 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.070922 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" containerName="istio-proxy" Oct 30 20:55:12.071019 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.071001 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" containerName="istio-proxy" Oct 30 20:55:12.071019 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.071008 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="feb15caf-387e-40ab-a8f8-d6292cf0b28b" containerName="istio-proxy" Oct 30 20:55:12.071019 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.071014 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="d983d7e5-28e1-44a2-82ba-f2cd4f8ca5e4" containerName="istio-proxy" Oct 30 20:55:12.071150 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.071025 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="feb15caf-387e-40ab-a8f8-d6292cf0b28b" containerName="istio-proxy" Oct 30 20:55:12.076110 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.076090 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m" Oct 30 20:55:12.081420 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.081397 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-eebcb4\"" Oct 30 20:55:12.087312 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.087285 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m"] Oct 30 20:55:12.099540 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.099511 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz"] Oct 30 20:55:12.100538 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.100512 4848 cpu_manager.go:414] "containerMap: removing container" podUID="feb15caf-387e-40ab-a8f8-d6292cf0b28b" containerName="istio-proxy" Oct 30 20:55:12.100538 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.100537 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feb15caf-387e-40ab-a8f8-d6292cf0b28b" containerName="istio-proxy" Oct 30 20:55:12.104940 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.104921 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz" Oct 30 20:55:12.111397 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.111371 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz"] Oct 30 20:55:12.111901 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.111878 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-eebcb4-dockercfg-mzjvq\"" Oct 30 20:55:12.133519 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.133484 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnhrg\" (UniqueName: \"kubernetes.io/projected/0a814981-8a9e-4c77-89f9-3b3423e1418c-kube-api-access-bnhrg\") pod \"payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz\" (UID: \"0a814981-8a9e-4c77-89f9-3b3423e1418c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz" Oct 30 20:55:12.133683 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.133559 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vgjx\" (UniqueName: \"kubernetes.io/projected/f9bd5631-bea1-41f0-a87c-9e3f83b0e688-kube-api-access-8vgjx\") pod \"payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m\" (UID: \"f9bd5631-bea1-41f0-a87c-9e3f83b0e688\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m" Oct 30 20:55:12.133683 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.133614 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a814981-8a9e-4c77-89f9-3b3423e1418c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz\" (UID: \"0a814981-8a9e-4c77-89f9-3b3423e1418c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz" Oct 30 20:55:12.133965 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.133944 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9bd5631-bea1-41f0-a87c-9e3f83b0e688-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m\" (UID: \"f9bd5631-bea1-41f0-a87c-9e3f83b0e688\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m" Oct 30 20:55:12.246790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.245946 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9bd5631-bea1-41f0-a87c-9e3f83b0e688-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m\" (UID: \"f9bd5631-bea1-41f0-a87c-9e3f83b0e688\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m" Oct 30 20:55:12.246790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.246085 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnhrg\" (UniqueName: \"kubernetes.io/projected/0a814981-8a9e-4c77-89f9-3b3423e1418c-kube-api-access-bnhrg\") pod \"payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz\" (UID: \"0a814981-8a9e-4c77-89f9-3b3423e1418c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz" Oct 30 20:55:12.246790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.246110 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vgjx\" (UniqueName: \"kubernetes.io/projected/f9bd5631-bea1-41f0-a87c-9e3f83b0e688-kube-api-access-8vgjx\") pod \"payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m\" (UID: \"f9bd5631-bea1-41f0-a87c-9e3f83b0e688\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m" Oct 30 20:55:12.246790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.246155 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a814981-8a9e-4c77-89f9-3b3423e1418c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz\" (UID: \"0a814981-8a9e-4c77-89f9-3b3423e1418c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz" Oct 30 20:55:12.262710 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.247176 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a814981-8a9e-4c77-89f9-3b3423e1418c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz\" (UID: \"0a814981-8a9e-4c77-89f9-3b3423e1418c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz" Oct 30 20:55:12.262710 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.247932 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9bd5631-bea1-41f0-a87c-9e3f83b0e688-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m\" (UID: \"f9bd5631-bea1-41f0-a87c-9e3f83b0e688\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m" Oct 30 20:55:12.303791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.299611 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vgjx\" (UniqueName: \"kubernetes.io/projected/f9bd5631-bea1-41f0-a87c-9e3f83b0e688-kube-api-access-8vgjx\") pod \"payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m\" (UID: \"f9bd5631-bea1-41f0-a87c-9e3f83b0e688\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m" Oct 30 20:55:12.308776 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.306719 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnhrg\" (UniqueName: \"kubernetes.io/projected/0a814981-8a9e-4c77-89f9-3b3423e1418c-kube-api-access-bnhrg\") pod \"payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz\" (UID: \"0a814981-8a9e-4c77-89f9-3b3423e1418c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz" Oct 30 20:55:12.409889 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.402035 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m" Oct 30 20:55:12.427711 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.427675 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz" Oct 30 20:55:12.767319 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.767288 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz"] Oct 30 20:55:12.767870 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:55:12.767690 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a814981_8a9e_4c77_89f9_3b3423e1418c.slice/crio-6e4cf15b879bb72ff56ec7e0c4d75f47e7c8dd041bc34444acb4d13d40ed7076 WatchSource:0}: Error finding container 6e4cf15b879bb72ff56ec7e0c4d75f47e7c8dd041bc34444acb4d13d40ed7076: Status 404 returned error can't find the container with id 6e4cf15b879bb72ff56ec7e0c4d75f47e7c8dd041bc34444acb4d13d40ed7076 Oct 30 20:55:12.771837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:12.771792 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m"] Oct 30 20:55:13.046856 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.046779 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh"] Oct 30 20:55:13.051323 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.051282 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh" Oct 30 20:55:13.057869 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.057844 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5e71c9\"" Oct 30 20:55:13.058030 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.058007 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5e71c9-dockercfg-jn2z8\"" Oct 30 20:55:13.060809 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.060760 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh"] Oct 30 20:55:13.065884 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.065854 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78f9485-256c-4073-be18-94be9a61eff2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh\" (UID: \"e78f9485-256c-4073-be18-94be9a61eff2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh" Oct 30 20:55:13.066004 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.065934 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjh4k\" (UniqueName: \"kubernetes.io/projected/e78f9485-256c-4073-be18-94be9a61eff2-kube-api-access-gjh4k\") pod \"payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh\" (UID: \"e78f9485-256c-4073-be18-94be9a61eff2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh" Oct 30 20:55:13.082160 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.082119 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn"] Oct 30 20:55:13.087557 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.087531 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn" Oct 30 20:55:13.094029 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.094001 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn"] Oct 30 20:55:13.179807 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.170739 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78f9485-256c-4073-be18-94be9a61eff2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh\" (UID: \"e78f9485-256c-4073-be18-94be9a61eff2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh" Oct 30 20:55:13.179807 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.170869 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjh4k\" (UniqueName: \"kubernetes.io/projected/e78f9485-256c-4073-be18-94be9a61eff2-kube-api-access-gjh4k\") pod \"payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh\" (UID: \"e78f9485-256c-4073-be18-94be9a61eff2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh" Oct 30 20:55:13.179807 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.173063 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78f9485-256c-4073-be18-94be9a61eff2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh\" (UID: \"e78f9485-256c-4073-be18-94be9a61eff2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh" Oct 30 20:55:13.233550 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.233515 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjh4k\" (UniqueName: \"kubernetes.io/projected/e78f9485-256c-4073-be18-94be9a61eff2-kube-api-access-gjh4k\") pod \"payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh\" (UID: \"e78f9485-256c-4073-be18-94be9a61eff2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh" Oct 30 20:55:13.284299 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.284252 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2p82\" (UniqueName: \"kubernetes.io/projected/94578ab9-2c80-4037-85c2-40f3143ebba1-kube-api-access-d2p82\") pod \"payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn\" (UID: \"94578ab9-2c80-4037-85c2-40f3143ebba1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn" Oct 30 20:55:13.284588 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.284563 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94578ab9-2c80-4037-85c2-40f3143ebba1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn\" (UID: \"94578ab9-2c80-4037-85c2-40f3143ebba1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn" Oct 30 20:55:13.375846 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.364013 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d" Oct 30 20:55:13.375846 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.365464 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g" Oct 30 20:55:13.388788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.379050 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh" Oct 30 20:55:13.394430 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.390537 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2p82\" (UniqueName: \"kubernetes.io/projected/94578ab9-2c80-4037-85c2-40f3143ebba1-kube-api-access-d2p82\") pod \"payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn\" (UID: \"94578ab9-2c80-4037-85c2-40f3143ebba1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn" Oct 30 20:55:13.394430 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.390594 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94578ab9-2c80-4037-85c2-40f3143ebba1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn\" (UID: \"94578ab9-2c80-4037-85c2-40f3143ebba1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn" Oct 30 20:55:13.415527 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.414855 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94578ab9-2c80-4037-85c2-40f3143ebba1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn\" (UID: \"94578ab9-2c80-4037-85c2-40f3143ebba1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn" Oct 30 20:55:13.469566 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.469499 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2p82\" (UniqueName: \"kubernetes.io/projected/94578ab9-2c80-4037-85c2-40f3143ebba1-kube-api-access-d2p82\") pod \"payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn\" (UID: \"94578ab9-2c80-4037-85c2-40f3143ebba1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn" Oct 30 20:55:13.602717 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.602682 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz" event={"ID":"0a814981-8a9e-4c77-89f9-3b3423e1418c","Type":"ContainerStarted","Data":"e791ee5848beb1cbc293866b66adf9332d9a88048af46c85e2afa64e29051eb0"} Oct 30 20:55:13.602949 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.602790 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz" event={"ID":"0a814981-8a9e-4c77-89f9-3b3423e1418c","Type":"ContainerStarted","Data":"6e4cf15b879bb72ff56ec7e0c4d75f47e7c8dd041bc34444acb4d13d40ed7076"} Oct 30 20:55:13.602949 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.602823 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz" Oct 30 20:55:13.605173 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.605146 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m" event={"ID":"f9bd5631-bea1-41f0-a87c-9e3f83b0e688","Type":"ContainerStarted","Data":"6753c36986ef4999a4b9df197536c1882e9f1142b110dd3df30bc62bfa2edb0c"} Oct 30 20:55:13.605377 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.605360 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m" event={"ID":"f9bd5631-bea1-41f0-a87c-9e3f83b0e688","Type":"ContainerStarted","Data":"719ce7530c32f0eafbfc317ea65e9051a5980f74ab1b53c365bba3ad34a04842"} Oct 30 20:55:13.605583 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.605569 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m" Oct 30 20:55:13.624990 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.624924 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz" podStartSLOduration=1.6249057649999998 podStartE2EDuration="1.624905765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:13.618843085 +0000 UTC m=+1018.798739593" watchObservedRunningTime="2026-10-30 20:55:13.624905765 +0000 UTC m=+1018.804802276" Oct 30 20:55:13.642756 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.642653 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m" podStartSLOduration=1.642639145 podStartE2EDuration="1.642639145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:13.639367395 +0000 UTC m=+1018.819263904" watchObservedRunningTime="2026-10-30 20:55:13.642639145 +0000 UTC m=+1018.822535652" Oct 30 20:55:13.713318 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.713242 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn" Oct 30 20:55:13.769482 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:13.769445 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh"] Oct 30 20:55:14.024659 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:14.024630 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn"] Oct 30 20:55:14.025759 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:55:14.025734 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94578ab9_2c80_4037_85c2_40f3143ebba1.slice/crio-fe077e1a5ecab5f61ae9d16b53379a9f62a973bc7a7aa17716e0b17f69ec72bc WatchSource:0}: Error finding container fe077e1a5ecab5f61ae9d16b53379a9f62a973bc7a7aa17716e0b17f69ec72bc: Status 404 returned error can't find the container with id fe077e1a5ecab5f61ae9d16b53379a9f62a973bc7a7aa17716e0b17f69ec72bc Oct 30 20:55:14.650277 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:14.650233 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh" event={"ID":"e78f9485-256c-4073-be18-94be9a61eff2","Type":"ContainerStarted","Data":"6071b707fdb58729e186635b17252702d85f220588361a54dfa3b16152fa2df2"} Oct 30 20:55:14.654475 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:14.654441 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh" event={"ID":"e78f9485-256c-4073-be18-94be9a61eff2","Type":"ContainerStarted","Data":"d6f5b2032d00f83c88b521d3cf5ab3ac5f3176a33be5cd416762b2ecb64a4be3"} Oct 30 20:55:14.654729 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:14.654688 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh" Oct 30 20:55:14.658081 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:14.657433 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn" event={"ID":"94578ab9-2c80-4037-85c2-40f3143ebba1","Type":"ContainerStarted","Data":"a2d0bb5d9f3ec75593968c6299afbfffd7cbba7db377915d3c673651b6d88944"} Oct 30 20:55:14.658081 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:14.657482 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn" event={"ID":"94578ab9-2c80-4037-85c2-40f3143ebba1","Type":"ContainerStarted","Data":"fe077e1a5ecab5f61ae9d16b53379a9f62a973bc7a7aa17716e0b17f69ec72bc"} Oct 30 20:55:14.658081 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:14.657998 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn" Oct 30 20:55:14.669848 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:14.669751 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh" podStartSLOduration=1.669733906 podStartE2EDuration="1.669733906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:14.669540497 +0000 UTC m=+1019.849437018" watchObservedRunningTime="2026-10-30 20:55:14.669733906 +0000 UTC m=+1019.849630416" Oct 30 20:55:14.685290 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:14.685236 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn" podStartSLOduration=1.685220245 podStartE2EDuration="1.685220245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:14.684233124 +0000 UTC m=+1019.864129631" watchObservedRunningTime="2026-10-30 20:55:14.685220245 +0000 UTC m=+1019.865116753" Oct 30 20:55:15.574523 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:15.574496 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" Oct 30 20:55:16.782860 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:16.782821 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh"] Oct 30 20:55:16.794792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:16.794719 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" Oct 30 20:55:16.797117 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:16.797092 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh"] Oct 30 20:55:16.801213 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:16.801171 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-eebcb4-serving-cert\"" Oct 30 20:55:16.994790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:16.985866 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/849dd3a0-afee-4409-be49-d0a264b2b31b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh\" (UID: \"849dd3a0-afee-4409-be49-d0a264b2b31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" Oct 30 20:55:16.994790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:16.986047 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/849dd3a0-afee-4409-be49-d0a264b2b31b-metrics-certs\") pod \"maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh\" (UID: \"849dd3a0-afee-4409-be49-d0a264b2b31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" Oct 30 20:55:16.994790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:16.986117 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p45m9\" (UniqueName: \"kubernetes.io/projected/849dd3a0-afee-4409-be49-d0a264b2b31b-kube-api-access-p45m9\") pod \"maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh\" (UID: \"849dd3a0-afee-4409-be49-d0a264b2b31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" Oct 30 20:55:16.994790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:16.986167 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/849dd3a0-afee-4409-be49-d0a264b2b31b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh\" (UID: \"849dd3a0-afee-4409-be49-d0a264b2b31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" Oct 30 20:55:17.087505 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:17.087426 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/849dd3a0-afee-4409-be49-d0a264b2b31b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh\" (UID: \"849dd3a0-afee-4409-be49-d0a264b2b31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" Oct 30 20:55:17.087661 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:17.087533 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/849dd3a0-afee-4409-be49-d0a264b2b31b-metrics-certs\") pod \"maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh\" (UID: \"849dd3a0-afee-4409-be49-d0a264b2b31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" Oct 30 20:55:17.087661 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:17.087611 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p45m9\" (UniqueName: \"kubernetes.io/projected/849dd3a0-afee-4409-be49-d0a264b2b31b-kube-api-access-p45m9\") pod \"maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh\" (UID: \"849dd3a0-afee-4409-be49-d0a264b2b31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" Oct 30 20:55:17.087661 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:17.087655 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/849dd3a0-afee-4409-be49-d0a264b2b31b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh\" (UID: \"849dd3a0-afee-4409-be49-d0a264b2b31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" Oct 30 20:55:17.088242 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:17.088214 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/849dd3a0-afee-4409-be49-d0a264b2b31b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh\" (UID: \"849dd3a0-afee-4409-be49-d0a264b2b31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" Oct 30 20:55:17.090127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:17.090107 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/849dd3a0-afee-4409-be49-d0a264b2b31b-metrics-certs\") pod \"maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh\" (UID: \"849dd3a0-afee-4409-be49-d0a264b2b31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" Oct 30 20:55:17.090235 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:17.090181 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/849dd3a0-afee-4409-be49-d0a264b2b31b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh\" (UID: \"849dd3a0-afee-4409-be49-d0a264b2b31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" Oct 30 20:55:17.103153 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:17.103118 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p45m9\" (UniqueName: \"kubernetes.io/projected/849dd3a0-afee-4409-be49-d0a264b2b31b-kube-api-access-p45m9\") pod \"maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh\" (UID: \"849dd3a0-afee-4409-be49-d0a264b2b31b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" Oct 30 20:55:17.115870 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:17.115846 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" Oct 30 20:55:17.398999 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:17.398971 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh"] Oct 30 20:55:17.400062 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:55:17.400039 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849dd3a0_afee_4409_be49_d0a264b2b31b.slice/crio-3df91fa4f683842525d3cb7855c025bdda8019afa04cb75fcb03402fd65d4499 WatchSource:0}: Error finding container 3df91fa4f683842525d3cb7855c025bdda8019afa04cb75fcb03402fd65d4499: Status 404 returned error can't find the container with id 3df91fa4f683842525d3cb7855c025bdda8019afa04cb75fcb03402fd65d4499 Oct 30 20:55:17.695081 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:17.695049 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" event={"ID":"849dd3a0-afee-4409-be49-d0a264b2b31b","Type":"ContainerStarted","Data":"3df91fa4f683842525d3cb7855c025bdda8019afa04cb75fcb03402fd65d4499"} Oct 30 20:55:17.958557 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:17.958484 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk"] Oct 30 20:55:17.965698 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:17.965313 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" Oct 30 20:55:17.974091 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:17.973942 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5e71c9-serving-cert\"" Oct 30 20:55:17.980338 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:17.980293 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk"] Oct 30 20:55:18.011139 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:18.011107 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxw8q\" (UniqueName: \"kubernetes.io/projected/07654e83-e9e1-4297-be2b-41591a932b39-kube-api-access-hxw8q\") pod \"maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk\" (UID: \"07654e83-e9e1-4297-be2b-41591a932b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" Oct 30 20:55:18.011313 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:18.011234 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07654e83-e9e1-4297-be2b-41591a932b39-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk\" (UID: \"07654e83-e9e1-4297-be2b-41591a932b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" Oct 30 20:55:18.011401 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:18.011381 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07654e83-e9e1-4297-be2b-41591a932b39-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk\" (UID: \"07654e83-e9e1-4297-be2b-41591a932b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" Oct 30 20:55:18.012165 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:18.012138 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07654e83-e9e1-4297-be2b-41591a932b39-metrics-certs\") pod \"maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk\" (UID: \"07654e83-e9e1-4297-be2b-41591a932b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" Oct 30 20:55:18.113490 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:18.113454 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07654e83-e9e1-4297-be2b-41591a932b39-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk\" (UID: \"07654e83-e9e1-4297-be2b-41591a932b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" Oct 30 20:55:18.113692 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:18.113559 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07654e83-e9e1-4297-be2b-41591a932b39-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk\" (UID: \"07654e83-e9e1-4297-be2b-41591a932b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" Oct 30 20:55:18.113692 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:18.113628 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07654e83-e9e1-4297-be2b-41591a932b39-metrics-certs\") pod \"maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk\" (UID: \"07654e83-e9e1-4297-be2b-41591a932b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" Oct 30 20:55:18.113692 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:18.113679 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxw8q\" (UniqueName: \"kubernetes.io/projected/07654e83-e9e1-4297-be2b-41591a932b39-kube-api-access-hxw8q\") pod \"maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk\" (UID: \"07654e83-e9e1-4297-be2b-41591a932b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" Oct 30 20:55:18.114265 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:18.114240 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07654e83-e9e1-4297-be2b-41591a932b39-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk\" (UID: \"07654e83-e9e1-4297-be2b-41591a932b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" Oct 30 20:55:18.115929 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:18.115907 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07654e83-e9e1-4297-be2b-41591a932b39-metrics-certs\") pod \"maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk\" (UID: \"07654e83-e9e1-4297-be2b-41591a932b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" Oct 30 20:55:18.116031 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:18.115945 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07654e83-e9e1-4297-be2b-41591a932b39-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk\" (UID: \"07654e83-e9e1-4297-be2b-41591a932b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" Oct 30 20:55:18.123219 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:18.123199 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxw8q\" (UniqueName: \"kubernetes.io/projected/07654e83-e9e1-4297-be2b-41591a932b39-kube-api-access-hxw8q\") pod \"maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk\" (UID: \"07654e83-e9e1-4297-be2b-41591a932b39\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" Oct 30 20:55:18.294553 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:18.293263 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" Oct 30 20:55:18.702038 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:18.698558 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk"] Oct 30 20:55:18.714360 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:18.714326 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" event={"ID":"849dd3a0-afee-4409-be49-d0a264b2b31b","Type":"ContainerStarted","Data":"5882201326f2cdb8865f807b883d2c7badff866e09ec5661edfc43d5e5615529"} Oct 30 20:55:18.715183 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:18.715166 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" Oct 30 20:55:18.717827 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:18.716732 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" event={"ID":"07654e83-e9e1-4297-be2b-41591a932b39","Type":"ContainerStarted","Data":"20951f7ee1140051b17e20a9595b1b66b12606a35bdd0192bb46835a055f13e8"} Oct 30 20:55:18.758861 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:18.757715 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" podStartSLOduration=2.279566398 podStartE2EDuration="2.75769155s" totalImagesPullingTime="478.125152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:18.736502484 +0000 UTC m=+1023.916398994" watchObservedRunningTime="2026-10-30 20:55:18.75769155 +0000 UTC m=+1023.937588061" Oct 30 20:55:19.725603 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:19.725570 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" event={"ID":"07654e83-e9e1-4297-be2b-41591a932b39","Type":"ContainerStarted","Data":"db59858308e17190b388ace19ca6b8414b1b9594369d12df322d9df456408080"} Oct 30 20:55:19.726049 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:19.725658 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" Oct 30 20:55:19.745328 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:19.745285 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" podStartSLOduration=2.299968329 podStartE2EDuration="2.745272214s" totalImagesPullingTime="445.303885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:19.741942695 +0000 UTC m=+1024.921839201" watchObservedRunningTime="2026-10-30 20:55:19.745272214 +0000 UTC m=+1024.925168756" Oct 30 20:55:21.191946 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.191915 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz"] Oct 30 20:55:21.196742 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.196719 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.210059 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.204905 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-7c14e2-openshift-default-dockercfg-qhj5m\"" Oct 30 20:55:21.216905 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.216876 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz"] Oct 30 20:55:21.284895 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.284859 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-credential-socket\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.284895 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.284900 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-envoy\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.285108 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.285001 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-workload-certs\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.285108 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.285037 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istiod-ca-cert\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.285108 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.285088 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-workload-socket\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.285217 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.285162 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmhcx\" (UniqueName: \"kubernetes.io/projected/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-kube-api-access-xmhcx\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.285217 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.285209 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-token\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.285358 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.285338 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-data\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.285485 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.285462 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-podinfo\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.392898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.392858 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istiod-ca-cert\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.394021 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.393996 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-workload-socket\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.409311 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.409280 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmhcx\" (UniqueName: \"kubernetes.io/projected/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-kube-api-access-xmhcx\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.409999 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.409972 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-token\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.410322 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.410303 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-data\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.410518 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.410502 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-podinfo\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.410685 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.410665 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-credential-socket\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.411877 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.411857 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-envoy\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.412064 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.412047 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-workload-certs\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.412336 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.411434 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-data\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.427266 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.393910 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istiod-ca-cert\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.427449 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.426049 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-workload-certs\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.427555 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.427169 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-credential-socket\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.427650 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.394456 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-workload-socket\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.438069 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.438041 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-podinfo\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.439012 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.438961 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmhcx\" (UniqueName: \"kubernetes.io/projected/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-kube-api-access-xmhcx\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.448054 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.447966 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-token\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.448424 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.448402 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-envoy\") pod \"e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.517195 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.516929 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:21.911972 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.911947 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz"] Oct 30 20:55:21.913158 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:55:21.913132 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e499bb_0b7e_4bb6_9966_2ac87b0c449f.slice/crio-3aa3b08ee995b1c220a22cbb0b223db60582aa5632e9aec64bce49fe55f6cc3b WatchSource:0}: Error finding container 3aa3b08ee995b1c220a22cbb0b223db60582aa5632e9aec64bce49fe55f6cc3b: Status 404 returned error can't find the container with id 3aa3b08ee995b1c220a22cbb0b223db60582aa5632e9aec64bce49fe55f6cc3b Oct 30 20:55:21.916121 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.916087 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:21.916206 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.916154 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:21.916206 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:21.916184 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:22.562855 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:55:22.562696 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:55:22.790099 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:22.790054 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" event={"ID":"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f","Type":"ContainerStarted","Data":"9743ef69836b78a654e81a7b5066df32e4c72851bd8fe21d78d647757db10a62"} Oct 30 20:55:22.790265 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:22.790130 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" event={"ID":"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f","Type":"ContainerStarted","Data":"3aa3b08ee995b1c220a22cbb0b223db60582aa5632e9aec64bce49fe55f6cc3b"} Oct 30 20:55:22.813875 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:22.813746 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" podStartSLOduration=1.813731687 podStartE2EDuration="1.813731687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:22.811679974 +0000 UTC m=+1027.991576481" watchObservedRunningTime="2026-10-30 20:55:22.813731687 +0000 UTC m=+1027.993628247" Oct 30 20:55:23.518787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:23.518225 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:23.541467 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:23.541358 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:23.798064 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:23.797990 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:23.799107 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:23.799087 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:55:24.661637 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:24.661607 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m" Oct 30 20:55:24.661822 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:24.661672 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz" Oct 30 20:55:25.669583 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:25.669550 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn" Oct 30 20:55:25.670103 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:25.669631 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh" Oct 30 20:55:25.738623 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:25.738569 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" Oct 30 20:55:25.739679 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:25.739659 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" Oct 30 20:55:30.923988 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:30.923951 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m"] Oct 30 20:55:30.971391 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:30.970736 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz"] Oct 30 20:55:31.271064 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:31.266972 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh"] Oct 30 20:55:31.273631 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:31.273604 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn"] Oct 30 20:55:35.925119 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:35.925061 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m" podUID="f9bd5631-bea1-41f0-a87c-9e3f83b0e688" containerName="payload-processing" containerID="cri-o://6753c36986ef4999a4b9df197536c1882e9f1142b110dd3df30bc62bfa2edb0c" gracePeriod=25 Oct 30 20:55:35.972000 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:35.971958 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz" podUID="0a814981-8a9e-4c77-89f9-3b3423e1418c" containerName="payload-pre-processing" containerID="cri-o://e791ee5848beb1cbc293866b66adf9332d9a88048af46c85e2afa64e29051eb0" gracePeriod=25 Oct 30 20:55:36.100911 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.100719 4848 generic.go:356] "Generic (PLEG): container finished" podID="0a814981-8a9e-4c77-89f9-3b3423e1418c" containerID="e791ee5848beb1cbc293866b66adf9332d9a88048af46c85e2afa64e29051eb0" exitCode=0 Oct 30 20:55:36.100911 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.100839 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz" event={"ID":"0a814981-8a9e-4c77-89f9-3b3423e1418c","Type":"ContainerDied","Data":"e791ee5848beb1cbc293866b66adf9332d9a88048af46c85e2afa64e29051eb0"} Oct 30 20:55:36.106413 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.106222 4848 generic.go:356] "Generic (PLEG): container finished" podID="f9bd5631-bea1-41f0-a87c-9e3f83b0e688" containerID="6753c36986ef4999a4b9df197536c1882e9f1142b110dd3df30bc62bfa2edb0c" exitCode=0 Oct 30 20:55:36.106413 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.106351 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m" event={"ID":"f9bd5631-bea1-41f0-a87c-9e3f83b0e688","Type":"ContainerDied","Data":"6753c36986ef4999a4b9df197536c1882e9f1142b110dd3df30bc62bfa2edb0c"} Oct 30 20:55:36.268059 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.268015 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh" podUID="e78f9485-256c-4073-be18-94be9a61eff2" containerName="payload-processing" containerID="cri-o://6071b707fdb58729e186635b17252702d85f220588361a54dfa3b16152fa2df2" gracePeriod=25 Oct 30 20:55:36.277325 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.276497 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn" podUID="94578ab9-2c80-4037-85c2-40f3143ebba1" containerName="payload-pre-processing" containerID="cri-o://a2d0bb5d9f3ec75593968c6299afbfffd7cbba7db377915d3c673651b6d88944" gracePeriod=25 Oct 30 20:55:36.449968 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.449936 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m" Oct 30 20:55:36.468482 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.468310 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz" Oct 30 20:55:36.536495 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.536454 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9bd5631-bea1-41f0-a87c-9e3f83b0e688-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9bd5631-bea1-41f0-a87c-9e3f83b0e688-plugins-config-volume\") pod \"f9bd5631-bea1-41f0-a87c-9e3f83b0e688\" (UID: \"f9bd5631-bea1-41f0-a87c-9e3f83b0e688\") " Oct 30 20:55:36.536830 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.536810 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a814981-8a9e-4c77-89f9-3b3423e1418c-kube-api-access-bnhrg\" (UniqueName: \"kubernetes.io/projected/0a814981-8a9e-4c77-89f9-3b3423e1418c-kube-api-access-bnhrg\") pod \"0a814981-8a9e-4c77-89f9-3b3423e1418c\" (UID: \"0a814981-8a9e-4c77-89f9-3b3423e1418c\") " Oct 30 20:55:36.536973 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.536958 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a814981-8a9e-4c77-89f9-3b3423e1418c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a814981-8a9e-4c77-89f9-3b3423e1418c-plugins-config-volume\") pod \"0a814981-8a9e-4c77-89f9-3b3423e1418c\" (UID: \"0a814981-8a9e-4c77-89f9-3b3423e1418c\") " Oct 30 20:55:36.537093 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.537079 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9bd5631-bea1-41f0-a87c-9e3f83b0e688-kube-api-access-8vgjx\" (UniqueName: \"kubernetes.io/projected/f9bd5631-bea1-41f0-a87c-9e3f83b0e688-kube-api-access-8vgjx\") pod \"f9bd5631-bea1-41f0-a87c-9e3f83b0e688\" (UID: \"f9bd5631-bea1-41f0-a87c-9e3f83b0e688\") " Oct 30 20:55:36.539153 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.538921 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a814981-8a9e-4c77-89f9-3b3423e1418c-plugins-config-volume" pod "0a814981-8a9e-4c77-89f9-3b3423e1418c" (UID: "0a814981-8a9e-4c77-89f9-3b3423e1418c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:36.539153 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.539123 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9bd5631-bea1-41f0-a87c-9e3f83b0e688-plugins-config-volume" pod "f9bd5631-bea1-41f0-a87c-9e3f83b0e688" (UID: "f9bd5631-bea1-41f0-a87c-9e3f83b0e688"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:36.559687 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.559648 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bd5631-bea1-41f0-a87c-9e3f83b0e688-kube-api-access-8vgjx" pod "f9bd5631-bea1-41f0-a87c-9e3f83b0e688" (UID: "f9bd5631-bea1-41f0-a87c-9e3f83b0e688"). InnerVolumeSpecName "kube-api-access-8vgjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:36.561426 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.561400 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a814981-8a9e-4c77-89f9-3b3423e1418c-kube-api-access-bnhrg" pod "0a814981-8a9e-4c77-89f9-3b3423e1418c" (UID: "0a814981-8a9e-4c77-89f9-3b3423e1418c"). InnerVolumeSpecName "kube-api-access-bnhrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:36.642484 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.642360 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9bd5631-bea1-41f0-a87c-9e3f83b0e688-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:36.642484 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.642401 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnhrg\" (UniqueName: \"kubernetes.io/projected/0a814981-8a9e-4c77-89f9-3b3423e1418c-kube-api-access-bnhrg\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:36.642484 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.642416 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a814981-8a9e-4c77-89f9-3b3423e1418c-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:36.642484 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.642430 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vgjx\" (UniqueName: \"kubernetes.io/projected/f9bd5631-bea1-41f0-a87c-9e3f83b0e688-kube-api-access-8vgjx\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:36.775155 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.775130 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh" Oct 30 20:55:36.824437 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.823523 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn" Oct 30 20:55:36.891632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.890922 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e78f9485-256c-4073-be18-94be9a61eff2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78f9485-256c-4073-be18-94be9a61eff2-plugins-config-volume\") pod \"e78f9485-256c-4073-be18-94be9a61eff2\" (UID: \"e78f9485-256c-4073-be18-94be9a61eff2\") " Oct 30 20:55:36.891632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.891347 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94578ab9-2c80-4037-85c2-40f3143ebba1-kube-api-access-d2p82\" (UniqueName: \"kubernetes.io/projected/94578ab9-2c80-4037-85c2-40f3143ebba1-kube-api-access-d2p82\") pod \"94578ab9-2c80-4037-85c2-40f3143ebba1\" (UID: \"94578ab9-2c80-4037-85c2-40f3143ebba1\") " Oct 30 20:55:36.893290 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.891995 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94578ab9-2c80-4037-85c2-40f3143ebba1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94578ab9-2c80-4037-85c2-40f3143ebba1-plugins-config-volume\") pod \"94578ab9-2c80-4037-85c2-40f3143ebba1\" (UID: \"94578ab9-2c80-4037-85c2-40f3143ebba1\") " Oct 30 20:55:36.893290 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.892132 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78f9485-256c-4073-be18-94be9a61eff2-kube-api-access-gjh4k\" (UniqueName: \"kubernetes.io/projected/e78f9485-256c-4073-be18-94be9a61eff2-kube-api-access-gjh4k\") pod \"e78f9485-256c-4073-be18-94be9a61eff2\" (UID: \"e78f9485-256c-4073-be18-94be9a61eff2\") " Oct 30 20:55:36.895551 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.895520 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78f9485-256c-4073-be18-94be9a61eff2-plugins-config-volume" pod "e78f9485-256c-4073-be18-94be9a61eff2" (UID: "e78f9485-256c-4073-be18-94be9a61eff2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:36.907985 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.903015 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94578ab9-2c80-4037-85c2-40f3143ebba1-plugins-config-volume" pod "94578ab9-2c80-4037-85c2-40f3143ebba1" (UID: "94578ab9-2c80-4037-85c2-40f3143ebba1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:36.917007 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.916963 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78f9485-256c-4073-be18-94be9a61eff2-kube-api-access-gjh4k" pod "e78f9485-256c-4073-be18-94be9a61eff2" (UID: "e78f9485-256c-4073-be18-94be9a61eff2"). InnerVolumeSpecName "kube-api-access-gjh4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:36.920130 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.920074 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94578ab9-2c80-4037-85c2-40f3143ebba1-kube-api-access-d2p82" pod "94578ab9-2c80-4037-85c2-40f3143ebba1" (UID: "94578ab9-2c80-4037-85c2-40f3143ebba1"). InnerVolumeSpecName "kube-api-access-d2p82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:36.995118 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.994758 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94578ab9-2c80-4037-85c2-40f3143ebba1-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:36.995118 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.994884 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjh4k\" (UniqueName: \"kubernetes.io/projected/e78f9485-256c-4073-be18-94be9a61eff2-kube-api-access-gjh4k\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:36.995118 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.994901 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78f9485-256c-4073-be18-94be9a61eff2-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:36.995118 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:36.994915 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2p82\" (UniqueName: \"kubernetes.io/projected/94578ab9-2c80-4037-85c2-40f3143ebba1-kube-api-access-d2p82\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:37.114794 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.114698 4848 generic.go:356] "Generic (PLEG): container finished" podID="e78f9485-256c-4073-be18-94be9a61eff2" containerID="6071b707fdb58729e186635b17252702d85f220588361a54dfa3b16152fa2df2" exitCode=0 Oct 30 20:55:37.114977 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.114761 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh" Oct 30 20:55:37.114977 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.114805 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh" event={"ID":"e78f9485-256c-4073-be18-94be9a61eff2","Type":"ContainerDied","Data":"6071b707fdb58729e186635b17252702d85f220588361a54dfa3b16152fa2df2"} Oct 30 20:55:37.114977 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.114879 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh" event={"ID":"e78f9485-256c-4073-be18-94be9a61eff2","Type":"ContainerDied","Data":"d6f5b2032d00f83c88b521d3cf5ab3ac5f3176a33be5cd416762b2ecb64a4be3"} Oct 30 20:55:37.114977 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.114916 4848 scope.go:122] "RemoveContainer" containerID="6071b707fdb58729e186635b17252702d85f220588361a54dfa3b16152fa2df2" Oct 30 20:55:37.118396 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.117748 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz" Oct 30 20:55:37.118396 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.117793 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz" event={"ID":"0a814981-8a9e-4c77-89f9-3b3423e1418c","Type":"ContainerDied","Data":"6e4cf15b879bb72ff56ec7e0c4d75f47e7c8dd041bc34444acb4d13d40ed7076"} Oct 30 20:55:37.121863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.121838 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m" event={"ID":"f9bd5631-bea1-41f0-a87c-9e3f83b0e688","Type":"ContainerDied","Data":"719ce7530c32f0eafbfc317ea65e9051a5980f74ab1b53c365bba3ad34a04842"} Oct 30 20:55:37.122026 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.121943 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m" Oct 30 20:55:37.126216 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.126018 4848 generic.go:356] "Generic (PLEG): container finished" podID="94578ab9-2c80-4037-85c2-40f3143ebba1" containerID="a2d0bb5d9f3ec75593968c6299afbfffd7cbba7db377915d3c673651b6d88944" exitCode=0 Oct 30 20:55:37.126216 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.126054 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn" Oct 30 20:55:37.126216 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.126092 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn" event={"ID":"94578ab9-2c80-4037-85c2-40f3143ebba1","Type":"ContainerDied","Data":"a2d0bb5d9f3ec75593968c6299afbfffd7cbba7db377915d3c673651b6d88944"} Oct 30 20:55:37.126216 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.126128 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn" event={"ID":"94578ab9-2c80-4037-85c2-40f3143ebba1","Type":"ContainerDied","Data":"fe077e1a5ecab5f61ae9d16b53379a9f62a973bc7a7aa17716e0b17f69ec72bc"} Oct 30 20:55:37.135915 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.135884 4848 scope.go:122] "RemoveContainer" containerID="6071b707fdb58729e186635b17252702d85f220588361a54dfa3b16152fa2df2" Oct 30 20:55:37.136211 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:55:37.136193 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6071b707fdb58729e186635b17252702d85f220588361a54dfa3b16152fa2df2\": container with ID starting with 6071b707fdb58729e186635b17252702d85f220588361a54dfa3b16152fa2df2 not found: ID does not exist" containerID="6071b707fdb58729e186635b17252702d85f220588361a54dfa3b16152fa2df2" Oct 30 20:55:37.136271 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.136219 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6071b707fdb58729e186635b17252702d85f220588361a54dfa3b16152fa2df2"} err="failed to get container status \"6071b707fdb58729e186635b17252702d85f220588361a54dfa3b16152fa2df2\": rpc error: code = NotFound desc = could not find container \"6071b707fdb58729e186635b17252702d85f220588361a54dfa3b16152fa2df2\": container with ID starting with 6071b707fdb58729e186635b17252702d85f220588361a54dfa3b16152fa2df2 not found: ID does not exist" Oct 30 20:55:37.136271 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.136234 4848 scope.go:122] "RemoveContainer" containerID="e791ee5848beb1cbc293866b66adf9332d9a88048af46c85e2afa64e29051eb0" Oct 30 20:55:37.140693 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.140668 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh"] Oct 30 20:55:37.146020 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.145999 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e71c9-68c969bc89-qszlh"] Oct 30 20:55:37.147279 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.147255 4848 scope.go:122] "RemoveContainer" containerID="6753c36986ef4999a4b9df197536c1882e9f1142b110dd3df30bc62bfa2edb0c" Oct 30 20:55:37.157582 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.157556 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m"] Oct 30 20:55:37.158462 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.158445 4848 scope.go:122] "RemoveContainer" containerID="a2d0bb5d9f3ec75593968c6299afbfffd7cbba7db377915d3c673651b6d88944" Oct 30 20:55:37.166163 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.166140 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-eebcb4-599fc8f499-xdc2m"] Oct 30 20:55:37.169192 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.169176 4848 scope.go:122] "RemoveContainer" containerID="a2d0bb5d9f3ec75593968c6299afbfffd7cbba7db377915d3c673651b6d88944" Oct 30 20:55:37.169554 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:55:37.169530 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d0bb5d9f3ec75593968c6299afbfffd7cbba7db377915d3c673651b6d88944\": container with ID starting with a2d0bb5d9f3ec75593968c6299afbfffd7cbba7db377915d3c673651b6d88944 not found: ID does not exist" containerID="a2d0bb5d9f3ec75593968c6299afbfffd7cbba7db377915d3c673651b6d88944" Oct 30 20:55:37.169640 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.169561 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d0bb5d9f3ec75593968c6299afbfffd7cbba7db377915d3c673651b6d88944"} err="failed to get container status \"a2d0bb5d9f3ec75593968c6299afbfffd7cbba7db377915d3c673651b6d88944\": rpc error: code = NotFound desc = could not find container \"a2d0bb5d9f3ec75593968c6299afbfffd7cbba7db377915d3c673651b6d88944\": container with ID starting with a2d0bb5d9f3ec75593968c6299afbfffd7cbba7db377915d3c673651b6d88944 not found: ID does not exist" Oct 30 20:55:37.184914 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.184883 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz"] Oct 30 20:55:37.194913 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.194886 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-eebcb4-6b98c99b7-vnmsz"] Oct 30 20:55:37.205890 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.205867 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn"] Oct 30 20:55:37.211756 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:37.211727 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e71c9-8444cbd59f-jmxtn"] Oct 30 20:55:39.043032 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:39.042994 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a814981-8a9e-4c77-89f9-3b3423e1418c" path="/var/lib/kubelet/pods/0a814981-8a9e-4c77-89f9-3b3423e1418c/volumes" Oct 30 20:55:39.043618 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:39.043600 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94578ab9-2c80-4037-85c2-40f3143ebba1" path="/var/lib/kubelet/pods/94578ab9-2c80-4037-85c2-40f3143ebba1/volumes" Oct 30 20:55:39.044225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:39.044202 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e78f9485-256c-4073-be18-94be9a61eff2" path="/var/lib/kubelet/pods/e78f9485-256c-4073-be18-94be9a61eff2/volumes" Oct 30 20:55:39.044750 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:39.044732 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9bd5631-bea1-41f0-a87c-9e3f83b0e688" path="/var/lib/kubelet/pods/f9bd5631-bea1-41f0-a87c-9e3f83b0e688/volumes" Oct 30 20:55:40.310715 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:40.310685 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g"] Oct 30 20:55:40.328820 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:40.327406 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d"] Oct 30 20:55:40.667891 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:40.651900 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:55:40.667891 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:40.651981 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" podUID="bf9525d3-a53f-4e30-8fae-12a407eace4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 20:55:40.771866 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:40.764295 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": read tcp 10.134.0.2:48800->10.134.0.105:15021: read: connection reset by peer" start-of-body= Oct 30 20:55:40.771866 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:40.764366 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" podUID="548c8714-1683-4286-865d-96d9d76cd5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": read tcp 10.134.0.2:48800->10.134.0.105:15021: read: connection reset by peer" Oct 30 20:55:40.840926 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:55:40.840882 4848 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548c8714_1683_4286_865d_96d9d76cd5ed.slice/crio-conmon-d965eb356a54058eb8f35428787584c97c196c994ae44c1d0ecb2679ba1d286d.scope\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@10-19851-0.service\": RecentStats: unable to find data in memory cache]" Oct 30 20:55:41.171175 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:41.171134 4848 generic.go:356] "Generic (PLEG): container finished" podID="548c8714-1683-4286-865d-96d9d76cd5ed" containerID="d965eb356a54058eb8f35428787584c97c196c994ae44c1d0ecb2679ba1d286d" exitCode=0 Oct 30 20:55:41.171372 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:41.171175 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" event={"ID":"548c8714-1683-4286-865d-96d9d76cd5ed","Type":"ContainerDied","Data":"d965eb356a54058eb8f35428787584c97c196c994ae44c1d0ecb2679ba1d286d"} Oct 30 20:55:41.171712 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:41.171675 4848 scope.go:122] "RemoveContainer" containerID="d965eb356a54058eb8f35428787584c97c196c994ae44c1d0ecb2679ba1d286d" Oct 30 20:55:41.172890 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:41.172869 4848 generic.go:356] "Generic (PLEG): container finished" podID="bf9525d3-a53f-4e30-8fae-12a407eace4e" containerID="1fe0afeb0b041b6f6c051e19cebc0b21d1e239f02ac85013d8ede4f6396dfe74" exitCode=0 Oct 30 20:55:41.172964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:41.172942 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" event={"ID":"bf9525d3-a53f-4e30-8fae-12a407eace4e","Type":"ContainerDied","Data":"1fe0afeb0b041b6f6c051e19cebc0b21d1e239f02ac85013d8ede4f6396dfe74"} Oct 30 20:55:41.173277 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:41.173264 4848 scope.go:122] "RemoveContainer" containerID="1fe0afeb0b041b6f6c051e19cebc0b21d1e239f02ac85013d8ede4f6396dfe74" Oct 30 20:55:41.173331 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:41.173307 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:41.173382 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:41.173370 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:41.173419 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:41.173402 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:41.174503 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:41.174479 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:41.174569 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:41.174534 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:41.174605 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:41.174566 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:41.503793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:41.503308 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4"] Oct 30 20:55:41.511875 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:41.507862 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" podUID="8e1e0e3a-b4f2-4537-848e-fdf5450dc36a" containerName="maas-api" containerID="cri-o://eac9c8ddddd077b77463b796add53b9dc3bfeba6551731eaad8286aabe196959" gracePeriod=30 Oct 30 20:55:42.132790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.132495 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" Oct 30 20:55:42.190255 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.190225 4848 generic.go:356] "Generic (PLEG): container finished" podID="8e1e0e3a-b4f2-4537-848e-fdf5450dc36a" containerID="eac9c8ddddd077b77463b796add53b9dc3bfeba6551731eaad8286aabe196959" exitCode=0 Oct 30 20:55:42.190437 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.190296 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" event={"ID":"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a","Type":"ContainerDied","Data":"eac9c8ddddd077b77463b796add53b9dc3bfeba6551731eaad8286aabe196959"} Oct 30 20:55:42.190437 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.190354 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" event={"ID":"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a","Type":"ContainerDied","Data":"3543da9085d884c2004622cdbe4c8008a46d081f1b4fa336f48862612671cf29"} Oct 30 20:55:42.190437 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.190384 4848 scope.go:122] "RemoveContainer" containerID="eac9c8ddddd077b77463b796add53b9dc3bfeba6551731eaad8286aabe196959" Oct 30 20:55:42.190616 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.190534 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4" Oct 30 20:55:42.192461 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.192439 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" event={"ID":"548c8714-1683-4286-865d-96d9d76cd5ed","Type":"ContainerStarted","Data":"580ecd10875162c4a8a6ba107cc05d2f486bdbf61dd9bc1af60b593af0abf961"} Oct 30 20:55:42.196188 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.196166 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" event={"ID":"bf9525d3-a53f-4e30-8fae-12a407eace4e","Type":"ContainerStarted","Data":"b5e2414fd2c098487de261474686523438e6a43c75b4887e369f2ebf909d5bd3"} Oct 30 20:55:42.213884 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.213519 4848 scope.go:122] "RemoveContainer" containerID="eac9c8ddddd077b77463b796add53b9dc3bfeba6551731eaad8286aabe196959" Oct 30 20:55:42.214221 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:55:42.214120 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eac9c8ddddd077b77463b796add53b9dc3bfeba6551731eaad8286aabe196959\": container with ID starting with eac9c8ddddd077b77463b796add53b9dc3bfeba6551731eaad8286aabe196959 not found: ID does not exist" containerID="eac9c8ddddd077b77463b796add53b9dc3bfeba6551731eaad8286aabe196959" Oct 30 20:55:42.214221 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.214157 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eac9c8ddddd077b77463b796add53b9dc3bfeba6551731eaad8286aabe196959"} err="failed to get container status \"eac9c8ddddd077b77463b796add53b9dc3bfeba6551731eaad8286aabe196959\": rpc error: code = NotFound desc = could not find container \"eac9c8ddddd077b77463b796add53b9dc3bfeba6551731eaad8286aabe196959\": container with ID starting with eac9c8ddddd077b77463b796add53b9dc3bfeba6551731eaad8286aabe196959 not found: ID does not exist" Oct 30 20:55:42.313800 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.311363 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-openshift-service-ca\") pod \"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a\" (UID: \"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a\") " Oct 30 20:55:42.313800 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.311420 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-metrics-certs\") pod \"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a\" (UID: \"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a\") " Oct 30 20:55:42.313800 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.311534 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-kube-api-access-5f9f8\" (UniqueName: \"kubernetes.io/projected/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-kube-api-access-5f9f8\") pod \"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a\" (UID: \"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a\") " Oct 30 20:55:42.313800 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.311651 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-maas-api-tls\") pod \"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a\" (UID: \"8e1e0e3a-b4f2-4537-848e-fdf5450dc36a\") " Oct 30 20:55:42.319879 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.319844 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-openshift-service-ca" pod "8e1e0e3a-b4f2-4537-848e-fdf5450dc36a" (UID: "8e1e0e3a-b4f2-4537-848e-fdf5450dc36a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:42.360870 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.343839 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-metrics-certs" pod "8e1e0e3a-b4f2-4537-848e-fdf5450dc36a" (UID: "8e1e0e3a-b4f2-4537-848e-fdf5450dc36a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:42.380937 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.380896 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-maas-api-tls" pod "8e1e0e3a-b4f2-4537-848e-fdf5450dc36a" (UID: "8e1e0e3a-b4f2-4537-848e-fdf5450dc36a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:55:42.381152 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.380922 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-kube-api-access-5f9f8" pod "8e1e0e3a-b4f2-4537-848e-fdf5450dc36a" (UID: "8e1e0e3a-b4f2-4537-848e-fdf5450dc36a"). InnerVolumeSpecName "kube-api-access-5f9f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:42.425793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.425045 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:42.425793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.425086 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:42.425793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.425101 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f9f8\" (UniqueName: \"kubernetes.io/projected/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-kube-api-access-5f9f8\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:42.425793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.425117 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:42.535793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.524027 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4"] Oct 30 20:55:42.535793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.529596 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a52bd2a1-79ccdd9df4-kfcs4"] Oct 30 20:55:42.821151 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.821000 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:42.832791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.824460 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:55:42.832791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.824523 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" podUID="bf9525d3-a53f-4e30-8fae-12a407eace4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 20:55:42.914790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.912993 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:42.914790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.914627 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 20:55:42.915048 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.914907 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" podUID="548c8714-1683-4286-865d-96d9d76cd5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 20:55:42.970198 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:42.970168 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e1e0e3a-b4f2-4537-848e-fdf5450dc36a" path="/var/lib/kubelet/pods/8e1e0e3a-b4f2-4537-848e-fdf5450dc36a/volumes" Oct 30 20:55:43.824790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:43.823129 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:55:43.824790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:43.823207 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" podUID="bf9525d3-a53f-4e30-8fae-12a407eace4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 20:55:43.926190 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:43.926154 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 20:55:43.926456 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:43.926429 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" podUID="548c8714-1683-4286-865d-96d9d76cd5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 20:55:44.823790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:44.820287 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:55:44.823790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:44.820355 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" podUID="bf9525d3-a53f-4e30-8fae-12a407eace4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 20:55:44.913141 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:44.913106 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 20:55:44.913734 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:44.913702 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" podUID="548c8714-1683-4286-865d-96d9d76cd5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 20:55:45.320792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:45.319857 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g" podUID="228304aa-a745-45d1-a305-d252cf89aff3" containerName="payload-processing" containerID="cri-o://7c4d4f823036297b997b69f37e75ee11697dba45aa4010d2c532373e86c3f10e" gracePeriod=25 Oct 30 20:55:45.335905 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:45.329001 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d" podUID="f6069037-4447-4cda-be07-3323e5cdf074" containerName="payload-pre-processing" containerID="cri-o://28642b2258c5c25c9c0701b44f680331a9775eddd7564db7a2435e39ff4dcd95" gracePeriod=25 Oct 30 20:55:45.824790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:45.821434 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:55:45.824790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:45.821492 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" podUID="bf9525d3-a53f-4e30-8fae-12a407eace4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 20:55:45.911790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:45.911200 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 20:55:45.911790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:45.911262 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" podUID="548c8714-1683-4286-865d-96d9d76cd5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 20:55:45.995500 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:45.995468 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g" Oct 30 20:55:45.998919 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:45.998900 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d" Oct 30 20:55:46.163065 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.162979 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6069037-4447-4cda-be07-3323e5cdf074-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6069037-4447-4cda-be07-3323e5cdf074-plugins-config-volume\") pod \"f6069037-4447-4cda-be07-3323e5cdf074\" (UID: \"f6069037-4447-4cda-be07-3323e5cdf074\") " Oct 30 20:55:46.163346 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.163325 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/228304aa-a745-45d1-a305-d252cf89aff3-kube-api-access-6rdzt\" (UniqueName: \"kubernetes.io/projected/228304aa-a745-45d1-a305-d252cf89aff3-kube-api-access-6rdzt\") pod \"228304aa-a745-45d1-a305-d252cf89aff3\" (UID: \"228304aa-a745-45d1-a305-d252cf89aff3\") " Oct 30 20:55:46.164602 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.164581 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6069037-4447-4cda-be07-3323e5cdf074-kube-api-access-rljpj\" (UniqueName: \"kubernetes.io/projected/f6069037-4447-4cda-be07-3323e5cdf074-kube-api-access-rljpj\") pod \"f6069037-4447-4cda-be07-3323e5cdf074\" (UID: \"f6069037-4447-4cda-be07-3323e5cdf074\") " Oct 30 20:55:46.164793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.164757 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/228304aa-a745-45d1-a305-d252cf89aff3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/228304aa-a745-45d1-a305-d252cf89aff3-plugins-config-volume\") pod \"228304aa-a745-45d1-a305-d252cf89aff3\" (UID: \"228304aa-a745-45d1-a305-d252cf89aff3\") " Oct 30 20:55:46.165913 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.165891 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228304aa-a745-45d1-a305-d252cf89aff3-plugins-config-volume" pod "228304aa-a745-45d1-a305-d252cf89aff3" (UID: "228304aa-a745-45d1-a305-d252cf89aff3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:46.166650 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.166512 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6069037-4447-4cda-be07-3323e5cdf074-plugins-config-volume" pod "f6069037-4447-4cda-be07-3323e5cdf074" (UID: "f6069037-4447-4cda-be07-3323e5cdf074"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:55:46.168068 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.168048 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228304aa-a745-45d1-a305-d252cf89aff3-kube-api-access-6rdzt" pod "228304aa-a745-45d1-a305-d252cf89aff3" (UID: "228304aa-a745-45d1-a305-d252cf89aff3"). InnerVolumeSpecName "kube-api-access-6rdzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:46.168862 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.168839 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6069037-4447-4cda-be07-3323e5cdf074-kube-api-access-rljpj" pod "f6069037-4447-4cda-be07-3323e5cdf074" (UID: "f6069037-4447-4cda-be07-3323e5cdf074"). InnerVolumeSpecName "kube-api-access-rljpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:55:46.266473 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.266437 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6069037-4447-4cda-be07-3323e5cdf074-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:46.266473 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.266468 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rdzt\" (UniqueName: \"kubernetes.io/projected/228304aa-a745-45d1-a305-d252cf89aff3-kube-api-access-6rdzt\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:46.266473 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.266477 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rljpj\" (UniqueName: \"kubernetes.io/projected/f6069037-4447-4cda-be07-3323e5cdf074-kube-api-access-rljpj\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:46.266727 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.266486 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/228304aa-a745-45d1-a305-d252cf89aff3-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:55:46.295474 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.295435 4848 generic.go:356] "Generic (PLEG): container finished" podID="228304aa-a745-45d1-a305-d252cf89aff3" containerID="7c4d4f823036297b997b69f37e75ee11697dba45aa4010d2c532373e86c3f10e" exitCode=0 Oct 30 20:55:46.295650 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.295486 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g" Oct 30 20:55:46.295650 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.295521 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g" event={"ID":"228304aa-a745-45d1-a305-d252cf89aff3","Type":"ContainerDied","Data":"7c4d4f823036297b997b69f37e75ee11697dba45aa4010d2c532373e86c3f10e"} Oct 30 20:55:46.295650 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.295643 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g" event={"ID":"228304aa-a745-45d1-a305-d252cf89aff3","Type":"ContainerDied","Data":"1edb6bcac502d35abacb14e601856a4f7f653c64b7d03eb148fc47aeacaa246c"} Oct 30 20:55:46.295803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.295678 4848 scope.go:122] "RemoveContainer" containerID="7c4d4f823036297b997b69f37e75ee11697dba45aa4010d2c532373e86c3f10e" Oct 30 20:55:46.297688 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.297652 4848 generic.go:356] "Generic (PLEG): container finished" podID="f6069037-4447-4cda-be07-3323e5cdf074" containerID="28642b2258c5c25c9c0701b44f680331a9775eddd7564db7a2435e39ff4dcd95" exitCode=0 Oct 30 20:55:46.297802 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.297788 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d" event={"ID":"f6069037-4447-4cda-be07-3323e5cdf074","Type":"ContainerDied","Data":"28642b2258c5c25c9c0701b44f680331a9775eddd7564db7a2435e39ff4dcd95"} Oct 30 20:55:46.297846 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.297825 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d" event={"ID":"f6069037-4447-4cda-be07-3323e5cdf074","Type":"ContainerDied","Data":"b197d0865b9413afc1873ac1878c4d7775cfd215c48de83120c9adaa2dbe2551"} Oct 30 20:55:46.297894 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.297877 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d" Oct 30 20:55:46.350113 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.350079 4848 scope.go:122] "RemoveContainer" containerID="7c4d4f823036297b997b69f37e75ee11697dba45aa4010d2c532373e86c3f10e" Oct 30 20:55:46.351035 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:55:46.351015 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c4d4f823036297b997b69f37e75ee11697dba45aa4010d2c532373e86c3f10e\": container with ID starting with 7c4d4f823036297b997b69f37e75ee11697dba45aa4010d2c532373e86c3f10e not found: ID does not exist" containerID="7c4d4f823036297b997b69f37e75ee11697dba45aa4010d2c532373e86c3f10e" Oct 30 20:55:46.351272 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.351253 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c4d4f823036297b997b69f37e75ee11697dba45aa4010d2c532373e86c3f10e"} err="failed to get container status \"7c4d4f823036297b997b69f37e75ee11697dba45aa4010d2c532373e86c3f10e\": rpc error: code = NotFound desc = could not find container \"7c4d4f823036297b997b69f37e75ee11697dba45aa4010d2c532373e86c3f10e\": container with ID starting with 7c4d4f823036297b997b69f37e75ee11697dba45aa4010d2c532373e86c3f10e not found: ID does not exist" Oct 30 20:55:46.351378 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.351366 4848 scope.go:122] "RemoveContainer" containerID="28642b2258c5c25c9c0701b44f680331a9775eddd7564db7a2435e39ff4dcd95" Oct 30 20:55:46.366545 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.366518 4848 scope.go:122] "RemoveContainer" containerID="28642b2258c5c25c9c0701b44f680331a9775eddd7564db7a2435e39ff4dcd95" Oct 30 20:55:46.367345 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:55:46.367320 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28642b2258c5c25c9c0701b44f680331a9775eddd7564db7a2435e39ff4dcd95\": container with ID starting with 28642b2258c5c25c9c0701b44f680331a9775eddd7564db7a2435e39ff4dcd95 not found: ID does not exist" containerID="28642b2258c5c25c9c0701b44f680331a9775eddd7564db7a2435e39ff4dcd95" Oct 30 20:55:46.367504 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.367482 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28642b2258c5c25c9c0701b44f680331a9775eddd7564db7a2435e39ff4dcd95"} err="failed to get container status \"28642b2258c5c25c9c0701b44f680331a9775eddd7564db7a2435e39ff4dcd95\": rpc error: code = NotFound desc = could not find container \"28642b2258c5c25c9c0701b44f680331a9775eddd7564db7a2435e39ff4dcd95\": container with ID starting with 28642b2258c5c25c9c0701b44f680331a9775eddd7564db7a2435e39ff4dcd95 not found: ID does not exist" Oct 30 20:55:46.371141 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.371121 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d"] Oct 30 20:55:46.378586 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.378567 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a52bd2a1-84d6b89f-ln64d"] Oct 30 20:55:46.389759 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.389736 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g"] Oct 30 20:55:46.394462 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.394441 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a52bd2a1-f96449b5f-mts7g"] Oct 30 20:55:46.821791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.821374 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:55:46.821791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.821438 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" podUID="bf9525d3-a53f-4e30-8fae-12a407eace4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 20:55:46.911755 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.911332 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 20:55:46.911755 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.911390 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" podUID="548c8714-1683-4286-865d-96d9d76cd5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 20:55:46.986133 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.986094 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="228304aa-a745-45d1-a305-d252cf89aff3" path="/var/lib/kubelet/pods/228304aa-a745-45d1-a305-d252cf89aff3/volumes" Oct 30 20:55:46.995723 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:46.995686 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6069037-4447-4cda-be07-3323e5cdf074" path="/var/lib/kubelet/pods/f6069037-4447-4cda-be07-3323e5cdf074/volumes" Oct 30 20:55:47.825791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:47.825052 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 20:55:47.825791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:47.825117 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" podUID="bf9525d3-a53f-4e30-8fae-12a407eace4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 20:55:47.923792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:47.922880 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 20:55:47.923792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:47.922952 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" podUID="548c8714-1683-4286-865d-96d9d76cd5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 20:55:48.914787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:48.912233 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 20:55:48.914787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:48.912302 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" podUID="548c8714-1683-4286-865d-96d9d76cd5ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 20:55:49.106232 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:49.106203 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:49.106667 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:49.106643 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:49.110247 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:49.108499 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:55:49.931675 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:49.931640 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:49.932245 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:49.932221 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:49.933154 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:49.933127 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:55:52.534607 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:55:52.534565 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:55:54.675237 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.675165 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5"] Oct 30 20:55:54.676626 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.676602 4848 cpu_manager.go:414] "containerMap: removing container" podUID="f6069037-4447-4cda-be07-3323e5cdf074" containerName="payload-pre-processing" Oct 30 20:55:54.676784 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.676756 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6069037-4447-4cda-be07-3323e5cdf074" containerName="payload-pre-processing" Oct 30 20:55:54.676916 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.676903 4848 cpu_manager.go:414] "containerMap: removing container" podUID="94578ab9-2c80-4037-85c2-40f3143ebba1" containerName="payload-pre-processing" Oct 30 20:55:54.677009 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.676997 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94578ab9-2c80-4037-85c2-40f3143ebba1" containerName="payload-pre-processing" Oct 30 20:55:54.677102 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.677092 4848 cpu_manager.go:414] "containerMap: removing container" podUID="228304aa-a745-45d1-a305-d252cf89aff3" containerName="payload-processing" Oct 30 20:55:54.677177 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.677166 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="228304aa-a745-45d1-a305-d252cf89aff3" containerName="payload-processing" Oct 30 20:55:54.677265 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.677255 4848 cpu_manager.go:414] "containerMap: removing container" podUID="8e1e0e3a-b4f2-4537-848e-fdf5450dc36a" containerName="maas-api" Oct 30 20:55:54.677345 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.677334 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1e0e3a-b4f2-4537-848e-fdf5450dc36a" containerName="maas-api" Oct 30 20:55:54.677429 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.677419 4848 cpu_manager.go:414] "containerMap: removing container" podUID="f9bd5631-bea1-41f0-a87c-9e3f83b0e688" containerName="payload-processing" Oct 30 20:55:54.677507 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.677496 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9bd5631-bea1-41f0-a87c-9e3f83b0e688" containerName="payload-processing" Oct 30 20:55:54.677607 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.677597 4848 cpu_manager.go:414] "containerMap: removing container" podUID="0a814981-8a9e-4c77-89f9-3b3423e1418c" containerName="payload-pre-processing" Oct 30 20:55:54.677686 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.677674 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a814981-8a9e-4c77-89f9-3b3423e1418c" containerName="payload-pre-processing" Oct 30 20:55:54.677783 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.677760 4848 cpu_manager.go:414] "containerMap: removing container" podUID="e78f9485-256c-4073-be18-94be9a61eff2" containerName="payload-processing" Oct 30 20:55:54.677873 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.677862 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78f9485-256c-4073-be18-94be9a61eff2" containerName="payload-processing" Oct 30 20:55:54.678113 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.678099 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9bd5631-bea1-41f0-a87c-9e3f83b0e688" containerName="payload-processing" Oct 30 20:55:54.678218 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.678207 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a814981-8a9e-4c77-89f9-3b3423e1418c" containerName="payload-pre-processing" Oct 30 20:55:54.678304 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.678294 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78f9485-256c-4073-be18-94be9a61eff2" containerName="payload-processing" Oct 30 20:55:54.678403 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.678392 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6069037-4447-4cda-be07-3323e5cdf074" containerName="payload-pre-processing" Oct 30 20:55:54.678491 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.678481 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="94578ab9-2c80-4037-85c2-40f3143ebba1" containerName="payload-pre-processing" Oct 30 20:55:54.678572 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.678561 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="228304aa-a745-45d1-a305-d252cf89aff3" containerName="payload-processing" Oct 30 20:55:54.678655 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.678643 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e1e0e3a-b4f2-4537-848e-fdf5450dc36a" containerName="maas-api" Oct 30 20:55:54.682331 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.682315 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.693359 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.693334 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-319188-openshift-default-dockercfg-lrf4c\"" Oct 30 20:55:54.700414 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.700381 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5"] Oct 30 20:55:54.738233 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.738193 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istiod-ca-cert\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.738233 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.738241 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qv9p\" (UniqueName: \"kubernetes.io/projected/86f1245b-8fd3-4831-abbd-fd9a428ce28b-kube-api-access-5qv9p\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.738504 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.738283 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-podinfo\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.738504 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.738328 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-credential-socket\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.738504 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.738376 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-workload-socket\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.738504 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.738398 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-workload-certs\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.738504 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.738414 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-data\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.738504 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.738457 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-token\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.738504 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.738481 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-envoy\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.795738 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.795703 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df"] Oct 30 20:55:54.799852 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.799831 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df" Oct 30 20:55:54.825792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.824987 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7c14e2-dockercfg-ngpdp\"" Oct 30 20:55:54.825792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.825706 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7c14e2\"" Oct 30 20:55:54.833539 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.833505 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df"] Oct 30 20:55:54.839468 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.839443 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-credential-socket\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.840029 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.840013 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-workload-socket\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.840354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.840342 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-workload-certs\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.840620 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.840608 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-data\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.841077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.841061 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-token\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.841436 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.841416 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-envoy\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.841686 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.841664 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istiod-ca-cert\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.842439 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.840956 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-data\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.842439 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.839929 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-credential-socket\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.842439 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.840581 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-workload-certs\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.842439 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.840287 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-workload-socket\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.843514 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.843317 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istiod-ca-cert\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.843704 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.843682 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qv9p\" (UniqueName: \"kubernetes.io/projected/86f1245b-8fd3-4831-abbd-fd9a428ce28b-kube-api-access-5qv9p\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.843943 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.843921 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-podinfo\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.846618 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.846591 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-podinfo\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.847923 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.847890 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-envoy\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.851220 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.851202 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-token\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:54.945535 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.945446 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a9974d-a731-4284-addd-58de76974f30-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df\" (UID: \"e1a9974d-a731-4284-addd-58de76974f30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df" Oct 30 20:55:54.945809 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:54.945786 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dxnj\" (UniqueName: \"kubernetes.io/projected/e1a9974d-a731-4284-addd-58de76974f30-kube-api-access-5dxnj\") pod \"payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df\" (UID: \"e1a9974d-a731-4284-addd-58de76974f30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df" Oct 30 20:55:55.006788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:55.005914 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qv9p\" (UniqueName: \"kubernetes.io/projected/86f1245b-8fd3-4831-abbd-fd9a428ce28b-kube-api-access-5qv9p\") pod \"e2e-api-a-319188-openshift-default-5bb45bd965-n24w5\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:55.042790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:55.040065 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:55.061359 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:55.049030 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a9974d-a731-4284-addd-58de76974f30-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df\" (UID: \"e1a9974d-a731-4284-addd-58de76974f30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df" Oct 30 20:55:55.061359 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:55.049097 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dxnj\" (UniqueName: \"kubernetes.io/projected/e1a9974d-a731-4284-addd-58de76974f30-kube-api-access-5dxnj\") pod \"payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df\" (UID: \"e1a9974d-a731-4284-addd-58de76974f30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df" Oct 30 20:55:55.061359 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:55.050607 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a9974d-a731-4284-addd-58de76974f30-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df\" (UID: \"e1a9974d-a731-4284-addd-58de76974f30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df" Oct 30 20:55:55.159125 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:55.159094 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dxnj\" (UniqueName: \"kubernetes.io/projected/e1a9974d-a731-4284-addd-58de76974f30-kube-api-access-5dxnj\") pod \"payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df\" (UID: \"e1a9974d-a731-4284-addd-58de76974f30\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df" Oct 30 20:55:55.449629 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:55.449599 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df" Oct 30 20:55:55.497312 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:55.497282 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5"] Oct 30 20:55:55.507345 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:55:55.507318 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f1245b_8fd3_4831_abbd_fd9a428ce28b.slice/crio-6789aef48f03b0e100816c5b10460daa7648601cda4bf4f55db97bcc0c90b4d3 WatchSource:0}: Error finding container 6789aef48f03b0e100816c5b10460daa7648601cda4bf4f55db97bcc0c90b4d3: Status 404 returned error can't find the container with id 6789aef48f03b0e100816c5b10460daa7648601cda4bf4f55db97bcc0c90b4d3 Oct 30 20:55:55.510803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:55.510752 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:55.510960 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:55.510837 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:55.510960 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:55.510869 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:55.848744 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:55.848707 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df"] Oct 30 20:55:56.503609 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:56.503569 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" event={"ID":"86f1245b-8fd3-4831-abbd-fd9a428ce28b","Type":"ContainerStarted","Data":"97c794d5d63b9970b3f77e92753010da1652eac92eee91ae2be5191082e80651"} Oct 30 20:55:56.503849 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:56.503625 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" event={"ID":"86f1245b-8fd3-4831-abbd-fd9a428ce28b","Type":"ContainerStarted","Data":"6789aef48f03b0e100816c5b10460daa7648601cda4bf4f55db97bcc0c90b4d3"} Oct 30 20:55:56.505720 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:56.505691 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df" event={"ID":"e1a9974d-a731-4284-addd-58de76974f30","Type":"ContainerStarted","Data":"9a16a1012cf687238f5145131ff690f9ce889231a0e521ea6c48f477ac0ed842"} Oct 30 20:55:56.527926 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:56.527877 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" podStartSLOduration=2.527862452 podStartE2EDuration="2.527862452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:56.523279856 +0000 UTC m=+1061.703176366" watchObservedRunningTime="2026-10-30 20:55:56.527862452 +0000 UTC m=+1061.707758959" Oct 30 20:55:57.040968 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:57.040934 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:57.047007 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:57.046970 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:57.518861 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:57.518818 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:57.519434 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:57.519415 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:55:58.261712 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.261671 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx"] Oct 30 20:55:58.269148 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.269120 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.274754 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.274727 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c5bf9a-openshift-default-dockercfg-chmgc\"" Oct 30 20:55:58.276212 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.276185 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx"] Oct 30 20:55:58.357855 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.357814 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn"] Oct 30 20:55:58.364212 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.364184 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.365064 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.365038 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-token\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.365219 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.365087 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istiod-ca-cert\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.365219 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.365180 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-workload-socket\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.365376 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.365263 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-credential-socket\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.365376 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.365296 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg6cp\" (UniqueName: \"kubernetes.io/projected/2d0f604c-f9f8-40fb-ab5e-8517520166a3-kube-api-access-vg6cp\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.365490 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.365455 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-podinfo\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.365549 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.365524 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-workload-certs\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.365636 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.365617 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-data\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.365717 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.365667 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-envoy\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.373961 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.373933 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn"] Oct 30 20:55:58.376208 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.376183 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a61855-openshift-default-dockercfg-4tm9w\"" Oct 30 20:55:58.468584 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.468547 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53b4de2-a49e-44d8-bc2e-369e52309561-istiod-ca-cert\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.468758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.468628 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-podinfo\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.468758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.468721 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-workload-certs\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.468918 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.468795 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-workload-certs\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.468918 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.468845 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-podinfo\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.469019 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.468939 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-envoy\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.469019 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.468970 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-workload-socket\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.469101 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.469034 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kvcs\" (UniqueName: \"kubernetes.io/projected/a53b4de2-a49e-44d8-bc2e-369e52309561-kube-api-access-6kvcs\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.469149 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.469127 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-data\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.469208 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.469185 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-envoy\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.469261 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.469207 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-workload-certs\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.469261 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.469240 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-token\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.469358 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.469288 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-token\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.469358 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.469325 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istiod-ca-cert\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.469481 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.469460 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-data\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.469558 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.469537 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-workload-socket\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.469623 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.469593 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-data\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.469678 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.469648 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-credential-socket\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.469678 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.469671 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-credential-socket\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.469841 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.469687 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg6cp\" (UniqueName: \"kubernetes.io/projected/2d0f604c-f9f8-40fb-ab5e-8517520166a3-kube-api-access-vg6cp\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.469919 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.469885 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-workload-socket\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.470075 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.470052 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istiod-ca-cert\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.470186 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.470165 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-credential-socket\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.471546 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.471524 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-podinfo\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.471654 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.471616 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-envoy\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.478663 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.478633 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-token\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.479313 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.479290 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg6cp\" (UniqueName: \"kubernetes.io/projected/2d0f604c-f9f8-40fb-ab5e-8517520166a3-kube-api-access-vg6cp\") pod \"e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.528017 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.527925 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df" event={"ID":"e1a9974d-a731-4284-addd-58de76974f30","Type":"ContainerStarted","Data":"19073d11ce0d283f236d328299f7a88e8f1b9d4738babacd6faeffb384aac9b2"} Oct 30 20:55:58.528332 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.528312 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df" Oct 30 20:55:58.546343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.546282 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df" podStartSLOduration=3.038524104 podStartE2EDuration="4.546264995s" totalImagesPullingTime="1.507740891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:58.545138566 +0000 UTC m=+1063.725035074" watchObservedRunningTime="2026-10-30 20:55:58.546264995 +0000 UTC m=+1063.726161503" Oct 30 20:55:58.578536 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.578494 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-credential-socket\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.578711 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.578608 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53b4de2-a49e-44d8-bc2e-369e52309561-istiod-ca-cert\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.578711 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.578669 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-workload-certs\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.578711 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.578704 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-podinfo\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.579308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.578820 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-envoy\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.579308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.578848 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-workload-socket\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.579308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.578873 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kvcs\" (UniqueName: \"kubernetes.io/projected/a53b4de2-a49e-44d8-bc2e-369e52309561-kube-api-access-6kvcs\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.579308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.579020 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-token\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.579308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.579141 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-credential-socket\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.579308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.579160 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-data\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.579655 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.579629 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-workload-socket\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.579730 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.579647 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-workload-certs\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.579914 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.579881 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53b4de2-a49e-44d8-bc2e-369e52309561-istiod-ca-cert\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.581134 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.581026 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-data\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.582250 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.582224 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-podinfo\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.583217 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.583195 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-envoy\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.587752 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.587725 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kvcs\" (UniqueName: \"kubernetes.io/projected/a53b4de2-a49e-44d8-bc2e-369e52309561-kube-api-access-6kvcs\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.588061 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.588025 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-token\") pod \"e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:58.599027 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.598999 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:58.693062 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:58.692759 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:55:59.072245 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:59.053299 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx"] Oct 30 20:55:59.161236 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:59.159696 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:59.161236 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:59.159809 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:59.161236 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:59.159852 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:59.217761 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:59.217733 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn"] Oct 30 20:55:59.222166 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:55:59.222131 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53b4de2_a49e_44d8_bc2e_369e52309561.slice/crio-5bbc03253a3d624d5dfd553d2fabd5b0df0494621a53c16c6f9143316b7cc4fe WatchSource:0}: Error finding container 5bbc03253a3d624d5dfd553d2fabd5b0df0494621a53c16c6f9143316b7cc4fe: Status 404 returned error can't find the container with id 5bbc03253a3d624d5dfd553d2fabd5b0df0494621a53c16c6f9143316b7cc4fe Oct 30 20:55:59.226176 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:59.226135 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:59.226279 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:59.226213 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:59.226279 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:59.226245 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:55:59.537916 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:59.537824 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" event={"ID":"a53b4de2-a49e-44d8-bc2e-369e52309561","Type":"ContainerStarted","Data":"de629f83777a61eb2754c58623a1973ca282be13d91288429c1233a95c42a060"} Oct 30 20:55:59.537916 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:59.537887 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" event={"ID":"a53b4de2-a49e-44d8-bc2e-369e52309561","Type":"ContainerStarted","Data":"5bbc03253a3d624d5dfd553d2fabd5b0df0494621a53c16c6f9143316b7cc4fe"} Oct 30 20:55:59.539699 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:59.539673 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" event={"ID":"2d0f604c-f9f8-40fb-ab5e-8517520166a3","Type":"ContainerStarted","Data":"8c4aa4c4e8cf753eeceea15dea7b0bc657490cd1de2b382fa629f05006d908b1"} Oct 30 20:55:59.539852 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:59.539716 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" event={"ID":"2d0f604c-f9f8-40fb-ab5e-8517520166a3","Type":"ContainerStarted","Data":"548ee4cd745ae21069ec48c1db1d35f30b9b0c34cc4220dd2e1f4f212885a2a4"} Oct 30 20:55:59.560635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:59.560584 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" podStartSLOduration=1.5605692580000001 podStartE2EDuration="1.560569258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:59.558482581 +0000 UTC m=+1064.738379101" watchObservedRunningTime="2026-10-30 20:55:59.560569258 +0000 UTC m=+1064.740465765" Oct 30 20:55:59.586007 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:59.585945 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" podStartSLOduration=1.5859265420000002 podStartE2EDuration="1.585926542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:55:59.584919646 +0000 UTC m=+1064.764816153" watchObservedRunningTime="2026-10-30 20:55:59.585926542 +0000 UTC m=+1064.765823050" Oct 30 20:55:59.599482 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:59.599400 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:55:59.694237 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:55:59.694209 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:56:00.226009 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:00.225809 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2"] Oct 30 20:56:00.230048 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:00.230020 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2" Oct 30 20:56:00.237397 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:00.237371 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-319188\"" Oct 30 20:56:00.240104 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:00.240080 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2"] Oct 30 20:56:00.385626 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:00.385550 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f9f3012-8124-49be-92c4-3f04feb2a82c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2\" (UID: \"9f9f3012-8124-49be-92c4-3f04feb2a82c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2" Oct 30 20:56:00.385853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:00.385681 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t56f\" (UniqueName: \"kubernetes.io/projected/9f9f3012-8124-49be-92c4-3f04feb2a82c-kube-api-access-5t56f\") pod \"payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2\" (UID: \"9f9f3012-8124-49be-92c4-3f04feb2a82c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2" Oct 30 20:56:00.494316 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:00.494214 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f9f3012-8124-49be-92c4-3f04feb2a82c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2\" (UID: \"9f9f3012-8124-49be-92c4-3f04feb2a82c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2" Oct 30 20:56:00.494757 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:00.494732 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t56f\" (UniqueName: \"kubernetes.io/projected/9f9f3012-8124-49be-92c4-3f04feb2a82c-kube-api-access-5t56f\") pod \"payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2\" (UID: \"9f9f3012-8124-49be-92c4-3f04feb2a82c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2" Oct 30 20:56:00.496304 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:00.496272 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f9f3012-8124-49be-92c4-3f04feb2a82c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2\" (UID: \"9f9f3012-8124-49be-92c4-3f04feb2a82c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2" Oct 30 20:56:00.524835 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:00.519934 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t56f\" (UniqueName: \"kubernetes.io/projected/9f9f3012-8124-49be-92c4-3f04feb2a82c-kube-api-access-5t56f\") pod \"payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2\" (UID: \"9f9f3012-8124-49be-92c4-3f04feb2a82c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2" Oct 30 20:56:00.589793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:00.586611 4848 generic.go:356] "Generic (PLEG): container finished" podID="b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" containerID="9743ef69836b78a654e81a7b5066df32e4c72851bd8fe21d78d647757db10a62" exitCode=0 Oct 30 20:56:00.589793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:00.587706 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" event={"ID":"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f","Type":"ContainerDied","Data":"9743ef69836b78a654e81a7b5066df32e4c72851bd8fe21d78d647757db10a62"} Oct 30 20:56:00.589793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:00.588158 4848 scope.go:122] "RemoveContainer" containerID="9743ef69836b78a654e81a7b5066df32e4c72851bd8fe21d78d647757db10a62" Oct 30 20:56:00.596788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:00.594953 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:00.596788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:00.595048 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:00.596788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:00.595104 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:00.633533 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:00.633501 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2" Oct 30 20:56:00.701086 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:00.701052 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:56:00.833722 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:00.831676 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:56:01.417488 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:01.417455 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2"] Oct 30 20:56:01.596851 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:01.596809 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" event={"ID":"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f","Type":"ContainerStarted","Data":"da481c7d2d6eeea25a8fc53a5bc200621989ada819d822d19f8ec19575eed0fb"} Oct 30 20:56:01.598640 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:01.598610 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2" event={"ID":"9f9f3012-8124-49be-92c4-3f04feb2a82c","Type":"ContainerStarted","Data":"98738c47a510fcd947330e9cd6dec89397e68a1bbdf0cd00473e65156866a153"} Oct 30 20:56:01.598757 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:01.598664 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2" event={"ID":"9f9f3012-8124-49be-92c4-3f04feb2a82c","Type":"ContainerStarted","Data":"8fb9bdad35fa2de029c41ca719725ec1746d808a6b7d6b634ccb8b6a191b3b11"} Oct 30 20:56:01.598971 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:01.598948 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:56:01.599055 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:01.598997 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2" Oct 30 20:56:01.599055 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:01.599018 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:56:01.599708 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:01.599690 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:56:01.600066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:01.600030 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:56:01.654215 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:01.654101 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2" podStartSLOduration=1.654085978 podStartE2EDuration="1.654085978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:01.651972355 +0000 UTC m=+1066.831868863" watchObservedRunningTime="2026-10-30 20:56:01.654085978 +0000 UTC m=+1066.833982511" Oct 30 20:56:02.525791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:02.525116 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:56:02.530056 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:02.530009 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:56:02.530220 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:02.530089 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" podUID="b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:56:03.528792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:03.528181 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:56:03.528792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:03.528251 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" podUID="b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:56:03.551108 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:03.551069 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl"] Oct 30 20:56:03.557283 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:03.557258 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl" Oct 30 20:56:03.565606 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:03.565580 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl"] Oct 30 20:56:03.566703 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:03.566682 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c5bf9a\"" Oct 30 20:56:03.567250 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:03.566926 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c5bf9a-dockercfg-vk5n8\"" Oct 30 20:56:03.686160 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:03.686121 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c0c1d0-a65b-47a4-a11b-30ec84f66b67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl\" (UID: \"c5c0c1d0-a65b-47a4-a11b-30ec84f66b67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl" Oct 30 20:56:03.693980 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:03.693929 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2csl2\" (UniqueName: \"kubernetes.io/projected/c5c0c1d0-a65b-47a4-a11b-30ec84f66b67-kube-api-access-2csl2\") pod \"payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl\" (UID: \"c5c0c1d0-a65b-47a4-a11b-30ec84f66b67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl" Oct 30 20:56:03.809978 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:03.809878 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c0c1d0-a65b-47a4-a11b-30ec84f66b67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl\" (UID: \"c5c0c1d0-a65b-47a4-a11b-30ec84f66b67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl" Oct 30 20:56:03.810386 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:03.810362 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2csl2\" (UniqueName: \"kubernetes.io/projected/c5c0c1d0-a65b-47a4-a11b-30ec84f66b67-kube-api-access-2csl2\") pod \"payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl\" (UID: \"c5c0c1d0-a65b-47a4-a11b-30ec84f66b67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl" Oct 30 20:56:03.846997 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:03.846954 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c0c1d0-a65b-47a4-a11b-30ec84f66b67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl\" (UID: \"c5c0c1d0-a65b-47a4-a11b-30ec84f66b67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl" Oct 30 20:56:04.037479 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:04.014498 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2csl2\" (UniqueName: \"kubernetes.io/projected/c5c0c1d0-a65b-47a4-a11b-30ec84f66b67-kube-api-access-2csl2\") pod \"payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl\" (UID: \"c5c0c1d0-a65b-47a4-a11b-30ec84f66b67\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl" Oct 30 20:56:04.175620 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:04.175205 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl" Oct 30 20:56:04.534849 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:04.526722 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:56:04.534849 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:04.526838 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" podUID="b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:56:04.549075 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:04.543119 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl"] Oct 30 20:56:04.549075 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:56:04.548147 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c0c1d0_a65b_47a4_a11b_30ec84f66b67.slice/crio-37201b57868ec231d480c398dbc04fc887c229cb4d55a15e1e9cf8dd9a90b30c WatchSource:0}: Error finding container 37201b57868ec231d480c398dbc04fc887c229cb4d55a15e1e9cf8dd9a90b30c: Status 404 returned error can't find the container with id 37201b57868ec231d480c398dbc04fc887c229cb4d55a15e1e9cf8dd9a90b30c Oct 30 20:56:04.663377 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:04.663339 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg"] Oct 30 20:56:04.675123 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:04.673488 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg" Oct 30 20:56:04.677776 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:04.677735 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg"] Oct 30 20:56:04.682508 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:04.682484 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a61855\"" Oct 30 20:56:04.682695 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:04.682508 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a61855-dockercfg-t2xmn\"" Oct 30 20:56:04.902839 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:04.877179 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4d9703d-5037-46a9-ad68-5b5b17618482-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg\" (UID: \"b4d9703d-5037-46a9-ad68-5b5b17618482\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg" Oct 30 20:56:04.902839 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:04.877325 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgl6n\" (UniqueName: \"kubernetes.io/projected/b4d9703d-5037-46a9-ad68-5b5b17618482-kube-api-access-jgl6n\") pod \"payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg\" (UID: \"b4d9703d-5037-46a9-ad68-5b5b17618482\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg" Oct 30 20:56:05.022789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:05.022659 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4d9703d-5037-46a9-ad68-5b5b17618482-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg\" (UID: \"b4d9703d-5037-46a9-ad68-5b5b17618482\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg" Oct 30 20:56:05.022789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:05.022788 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgl6n\" (UniqueName: \"kubernetes.io/projected/b4d9703d-5037-46a9-ad68-5b5b17618482-kube-api-access-jgl6n\") pod \"payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg\" (UID: \"b4d9703d-5037-46a9-ad68-5b5b17618482\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg" Oct 30 20:56:05.039266 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:05.024273 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4d9703d-5037-46a9-ad68-5b5b17618482-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg\" (UID: \"b4d9703d-5037-46a9-ad68-5b5b17618482\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg" Oct 30 20:56:05.097063 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:05.097027 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgl6n\" (UniqueName: \"kubernetes.io/projected/b4d9703d-5037-46a9-ad68-5b5b17618482-kube-api-access-jgl6n\") pod \"payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg\" (UID: \"b4d9703d-5037-46a9-ad68-5b5b17618482\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg" Oct 30 20:56:05.143798 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:05.140591 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl" event={"ID":"c5c0c1d0-a65b-47a4-a11b-30ec84f66b67","Type":"ContainerStarted","Data":"cfa407f945b6d61a00e298a682f009babdfb638c8b46896d9ca932e56038bbca"} Oct 30 20:56:05.143798 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:05.140653 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl" event={"ID":"c5c0c1d0-a65b-47a4-a11b-30ec84f66b67","Type":"ContainerStarted","Data":"37201b57868ec231d480c398dbc04fc887c229cb4d55a15e1e9cf8dd9a90b30c"} Oct 30 20:56:05.144243 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:05.144217 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl" Oct 30 20:56:05.204115 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:05.204052 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl" podStartSLOduration=2.204029825 podStartE2EDuration="2.204029825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:05.199130575 +0000 UTC m=+1070.379027084" watchObservedRunningTime="2026-10-30 20:56:05.204029825 +0000 UTC m=+1070.383926335" Oct 30 20:56:05.318789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:05.311049 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg" Oct 30 20:56:05.539285 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:05.538819 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:56:05.539285 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:05.538871 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" podUID="b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:56:05.748338 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:56:05.748305 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4d9703d_5037_46a9_ad68_5b5b17618482.slice/crio-d97aef9171b33cdddda81184186ca689f5af1236062f1971b85942d6ce45ee4b WatchSource:0}: Error finding container d97aef9171b33cdddda81184186ca689f5af1236062f1971b85942d6ce45ee4b: Status 404 returned error can't find the container with id d97aef9171b33cdddda81184186ca689f5af1236062f1971b85942d6ce45ee4b Oct 30 20:56:05.749434 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:05.749408 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg"] Oct 30 20:56:06.258791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:06.255980 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg" event={"ID":"b4d9703d-5037-46a9-ad68-5b5b17618482","Type":"ContainerStarted","Data":"882cc6918378c7a29f8dcab0ce0d6a6806e34143adc078a207465547540754a7"} Oct 30 20:56:06.258791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:06.256083 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg" event={"ID":"b4d9703d-5037-46a9-ad68-5b5b17618482","Type":"ContainerStarted","Data":"d97aef9171b33cdddda81184186ca689f5af1236062f1971b85942d6ce45ee4b"} Oct 30 20:56:06.258791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:06.256153 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg" Oct 30 20:56:06.282863 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:06.282810 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg" podStartSLOduration=2.282796543 podStartE2EDuration="2.282796543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:06.281400914 +0000 UTC m=+1071.461297422" watchObservedRunningTime="2026-10-30 20:56:06.282796543 +0000 UTC m=+1071.462693049" Oct 30 20:56:06.522849 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:06.521179 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:56:06.522849 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:06.521238 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" podUID="b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:56:07.520185 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:07.520148 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 20:56:07.520807 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:07.520755 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" podUID="b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 20:56:08.523211 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:08.523179 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:56:08.523603 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:08.523578 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:56:08.524448 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:08.524428 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:56:09.109484 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:09.109444 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz"] Oct 30 20:56:09.115610 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:09.115585 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" Oct 30 20:56:09.126423 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:09.126104 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-319188-serving-cert\"" Oct 30 20:56:09.135053 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:09.135002 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz"] Oct 30 20:56:09.295660 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:09.283689 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd50e1a1-a095-4b6a-a679-821ef4d48d41-metrics-certs\") pod \"maas-api-e2e-api-a-319188-7b6bcd5789-prptz\" (UID: \"bd50e1a1-a095-4b6a-a679-821ef4d48d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" Oct 30 20:56:09.296246 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:09.296209 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc595\" (UniqueName: \"kubernetes.io/projected/bd50e1a1-a095-4b6a-a679-821ef4d48d41-kube-api-access-hc595\") pod \"maas-api-e2e-api-a-319188-7b6bcd5789-prptz\" (UID: \"bd50e1a1-a095-4b6a-a679-821ef4d48d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" Oct 30 20:56:09.296499 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:09.296481 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd50e1a1-a095-4b6a-a679-821ef4d48d41-maas-api-tls\") pod \"maas-api-e2e-api-a-319188-7b6bcd5789-prptz\" (UID: \"bd50e1a1-a095-4b6a-a679-821ef4d48d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" Oct 30 20:56:09.296722 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:09.296707 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd50e1a1-a095-4b6a-a679-821ef4d48d41-openshift-service-ca\") pod \"maas-api-e2e-api-a-319188-7b6bcd5789-prptz\" (UID: \"bd50e1a1-a095-4b6a-a679-821ef4d48d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" Oct 30 20:56:09.398310 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:09.398222 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd50e1a1-a095-4b6a-a679-821ef4d48d41-openshift-service-ca\") pod \"maas-api-e2e-api-a-319188-7b6bcd5789-prptz\" (UID: \"bd50e1a1-a095-4b6a-a679-821ef4d48d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" Oct 30 20:56:09.398492 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:09.398416 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd50e1a1-a095-4b6a-a679-821ef4d48d41-metrics-certs\") pod \"maas-api-e2e-api-a-319188-7b6bcd5789-prptz\" (UID: \"bd50e1a1-a095-4b6a-a679-821ef4d48d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" Oct 30 20:56:09.398568 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:09.398530 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc595\" (UniqueName: \"kubernetes.io/projected/bd50e1a1-a095-4b6a-a679-821ef4d48d41-kube-api-access-hc595\") pod \"maas-api-e2e-api-a-319188-7b6bcd5789-prptz\" (UID: \"bd50e1a1-a095-4b6a-a679-821ef4d48d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" Oct 30 20:56:09.398568 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:09.398558 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd50e1a1-a095-4b6a-a679-821ef4d48d41-maas-api-tls\") pod \"maas-api-e2e-api-a-319188-7b6bcd5789-prptz\" (UID: \"bd50e1a1-a095-4b6a-a679-821ef4d48d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" Oct 30 20:56:09.399191 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:09.399158 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd50e1a1-a095-4b6a-a679-821ef4d48d41-openshift-service-ca\") pod \"maas-api-e2e-api-a-319188-7b6bcd5789-prptz\" (UID: \"bd50e1a1-a095-4b6a-a679-821ef4d48d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" Oct 30 20:56:09.401006 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:09.400983 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd50e1a1-a095-4b6a-a679-821ef4d48d41-metrics-certs\") pod \"maas-api-e2e-api-a-319188-7b6bcd5789-prptz\" (UID: \"bd50e1a1-a095-4b6a-a679-821ef4d48d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" Oct 30 20:56:09.401133 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:09.401035 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd50e1a1-a095-4b6a-a679-821ef4d48d41-maas-api-tls\") pod \"maas-api-e2e-api-a-319188-7b6bcd5789-prptz\" (UID: \"bd50e1a1-a095-4b6a-a679-821ef4d48d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" Oct 30 20:56:09.407283 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:09.407256 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc595\" (UniqueName: \"kubernetes.io/projected/bd50e1a1-a095-4b6a-a679-821ef4d48d41-kube-api-access-hc595\") pod \"maas-api-e2e-api-a-319188-7b6bcd5789-prptz\" (UID: \"bd50e1a1-a095-4b6a-a679-821ef4d48d41\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" Oct 30 20:56:09.439436 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:09.439406 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" Oct 30 20:56:09.543378 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:09.542435 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df" Oct 30 20:56:09.721680 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:56:09.721651 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd50e1a1_a095_4b6a_a679_821ef4d48d41.slice/crio-647abdf9b270bd98ed797afa31c4a0a054bff046452e379a455ed685265cf957 WatchSource:0}: Error finding container 647abdf9b270bd98ed797afa31c4a0a054bff046452e379a455ed685265cf957: Status 404 returned error can't find the container with id 647abdf9b270bd98ed797afa31c4a0a054bff046452e379a455ed685265cf957 Oct 30 20:56:09.721680 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:09.721659 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz"] Oct 30 20:56:10.336902 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:10.336865 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" event={"ID":"bd50e1a1-a095-4b6a-a679-821ef4d48d41","Type":"ContainerStarted","Data":"647abdf9b270bd98ed797afa31c4a0a054bff046452e379a455ed685265cf957"} Oct 30 20:56:11.346575 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:11.346538 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" event={"ID":"bd50e1a1-a095-4b6a-a679-821ef4d48d41","Type":"ContainerStarted","Data":"e8afbc4a6e4ac71feaea31aaca439e1537865b70eb1733171e16051ffe4e692e"} Oct 30 20:56:11.347051 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:11.346639 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" Oct 30 20:56:11.364175 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:11.364119 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" podStartSLOduration=1.8885274010000002 podStartE2EDuration="2.364101095s" totalImagesPullingTime="475.573694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:11.363138303 +0000 UTC m=+1076.543034830" watchObservedRunningTime="2026-10-30 20:56:11.364101095 +0000 UTC m=+1076.543997621" Oct 30 20:56:12.280221 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.280183 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws"] Oct 30 20:56:12.285993 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.285967 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.294461 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.294434 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-351fd1-openshift-default-dockercfg-b2pln\"" Oct 30 20:56:12.298963 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.298940 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws"] Oct 30 20:56:12.467066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.467024 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkz59\" (UniqueName: \"kubernetes.io/projected/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-kube-api-access-zkz59\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.467066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.467070 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-envoy\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.467557 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.467287 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-workload-socket\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.467557 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.467335 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istiod-ca-cert\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.467557 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.467436 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-podinfo\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.467557 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.467479 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-token\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.467557 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.467524 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-credential-socket\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.467855 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.467830 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-workload-certs\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.467972 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.467954 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-data\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.572853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.571894 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkz59\" (UniqueName: \"kubernetes.io/projected/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-kube-api-access-zkz59\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.572853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.571954 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-envoy\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.572853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.572105 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-workload-socket\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.572853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.572163 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istiod-ca-cert\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.572853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.572260 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-podinfo\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.572853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.572324 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-token\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.572853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.572379 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-credential-socket\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.572853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.572461 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-workload-certs\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.572853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.572579 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-data\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.573450 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.573141 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-data\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.580792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.579129 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-credential-socket\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.580792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.579693 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-workload-certs\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.580792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.579958 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-workload-socket\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.580792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.580561 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istiod-ca-cert\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.647376 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.639840 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-podinfo\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.647376 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.647294 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-token\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.652789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.650687 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-envoy\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.652789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.651436 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkz59\" (UniqueName: \"kubernetes.io/projected/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-kube-api-access-zkz59\") pod \"e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:12.702815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.700462 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2" Oct 30 20:56:12.920486 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:12.920411 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:13.321781 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:13.320663 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws"] Oct 30 20:56:13.329043 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:13.328593 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:13.329043 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:13.328662 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:13.329043 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:13.328701 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:13.381397 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:13.381367 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" event={"ID":"0c4e64ec-eb49-4a9e-86df-7483d597fd9c","Type":"ContainerStarted","Data":"e788e737f30b2178c9a6622d638db0f8a4db8d6fc7c2b49332dd770ceb564284"} Oct 30 20:56:14.391014 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:14.390980 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" event={"ID":"0c4e64ec-eb49-4a9e-86df-7483d597fd9c","Type":"ContainerStarted","Data":"95b6dc03d99fc0248e0083fa8cd12911f66e177064eb0980fd6b9ba309ba5fcc"} Oct 30 20:56:14.418997 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:14.418947 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" podStartSLOduration=2.418933049 podStartE2EDuration="2.418933049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:14.413945782 +0000 UTC m=+1079.593842293" watchObservedRunningTime="2026-10-30 20:56:14.418933049 +0000 UTC m=+1079.598829556" Oct 30 20:56:14.922993 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:14.922960 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:14.935004 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:14.934976 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:15.398428 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:15.398383 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:15.399463 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:15.399439 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:56:17.271376 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:17.271346 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl" Oct 30 20:56:17.274098 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:17.274070 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg" Oct 30 20:56:17.375452 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:17.375422 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" Oct 30 20:56:22.511384 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:56:22.511342 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:56:25.255556 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:25.253256 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn"] Oct 30 20:56:25.355872 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:25.352569 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn"] Oct 30 20:56:25.355872 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:25.352704 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:25.398799 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:25.390109 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-5452c3-openshift-default-dockercfg-gc6hf\"" Oct 30 20:56:25.721205 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:25.721163 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-credential-socket\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:25.721640 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:25.721610 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-token\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:25.721835 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:25.721814 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-workload-certs\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:25.725913 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:25.725884 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-podinfo\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:25.833743 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:25.823655 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-workload-socket\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:25.839880 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:25.834076 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8bgx\" (UniqueName: \"kubernetes.io/projected/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-kube-api-access-t8bgx\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:25.855800 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:25.840222 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-envoy\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:25.856154 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:25.856118 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istiod-ca-cert\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:25.856497 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:25.856479 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-data\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:26.030817 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.015859 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-workload-certs\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:26.031206 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.031175 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-podinfo\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:26.031354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.031338 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-workload-socket\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:26.031524 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.031510 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8bgx\" (UniqueName: \"kubernetes.io/projected/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-kube-api-access-t8bgx\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:26.045806 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.039915 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-envoy\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:26.045806 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.040045 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istiod-ca-cert\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:26.045806 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.040214 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-data\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:26.045806 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.040322 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-credential-socket\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:26.045806 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.040422 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-token\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:26.045806 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.024225 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-workload-certs\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:26.045806 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.044930 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-workload-socket\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:26.077790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.049537 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istiod-ca-cert\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:26.077790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.063930 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-credential-socket\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:26.077790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.064755 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-data\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:26.087787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.078687 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-podinfo\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:26.096006 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.094457 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-token\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:26.096520 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.096483 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-envoy\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:26.097484 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.097428 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8bgx\" (UniqueName: \"kubernetes.io/projected/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-kube-api-access-t8bgx\") pod \"e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:26.372916 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.372827 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:26.758304 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.758260 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn"] Oct 30 20:56:26.763405 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:56:26.763373 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3e6ed8_9d1e_4716_ab89_a37141dc030d.slice/crio-d497f3c6c09f4db3061fe658200d5f3b3e1a5239ac337e6c16bfaa435cfe4794 WatchSource:0}: Error finding container d497f3c6c09f4db3061fe658200d5f3b3e1a5239ac337e6c16bfaa435cfe4794: Status 404 returned error can't find the container with id d497f3c6c09f4db3061fe658200d5f3b3e1a5239ac337e6c16bfaa435cfe4794 Oct 30 20:56:26.765821 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.765785 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:26.765938 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.765856 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:26.765938 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.765888 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:26.892793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:26.891862 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" event={"ID":"3f3e6ed8-9d1e-4716-ab89-a37141dc030d","Type":"ContainerStarted","Data":"d497f3c6c09f4db3061fe658200d5f3b3e1a5239ac337e6c16bfaa435cfe4794"} Oct 30 20:56:27.754016 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:27.753980 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl"] Oct 30 20:56:28.101849 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:28.099521 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" event={"ID":"3f3e6ed8-9d1e-4716-ab89-a37141dc030d","Type":"ContainerStarted","Data":"29be838c4e202a929f01d3ae03df0128512314e8de2d5fd95bea31235a40a046"} Oct 30 20:56:28.199753 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:28.199652 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" podStartSLOduration=3.1996300189999998 podStartE2EDuration="3.199630019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:28.12427456 +0000 UTC m=+1093.304171095" watchObservedRunningTime="2026-10-30 20:56:28.199630019 +0000 UTC m=+1093.379526529" Oct 30 20:56:28.380368 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:28.380278 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:28.380722 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:28.380697 4848 patch_prober.go:29] interesting pod/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 20:56:28.380914 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:28.380889 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" podUID="3f3e6ed8-9d1e-4716-ab89-a37141dc030d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 20:56:29.379272 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:29.379245 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:30.153193 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:30.153162 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:30.159838 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:30.158894 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:56:32.818365 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:32.813484 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl" podUID="c5c0c1d0-a65b-47a4-a11b-30ec84f66b67" containerName="payload-pre-processing" containerID="cri-o://cfa407f945b6d61a00e298a682f009babdfb638c8b46896d9ca932e56038bbca" gracePeriod=25 Oct 30 20:56:32.845790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:32.843250 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg"] Oct 30 20:56:34.232812 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:34.232745 4848 generic.go:356] "Generic (PLEG): container finished" podID="c5c0c1d0-a65b-47a4-a11b-30ec84f66b67" containerID="cfa407f945b6d61a00e298a682f009babdfb638c8b46896d9ca932e56038bbca" exitCode=0 Oct 30 20:56:34.233274 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:34.232839 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl" event={"ID":"c5c0c1d0-a65b-47a4-a11b-30ec84f66b67","Type":"ContainerDied","Data":"cfa407f945b6d61a00e298a682f009babdfb638c8b46896d9ca932e56038bbca"} Oct 30 20:56:34.278708 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:34.278675 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl" Oct 30 20:56:34.292338 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:34.292297 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c0c1d0-a65b-47a4-a11b-30ec84f66b67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c0c1d0-a65b-47a4-a11b-30ec84f66b67-plugins-config-volume\") pod \"c5c0c1d0-a65b-47a4-a11b-30ec84f66b67\" (UID: \"c5c0c1d0-a65b-47a4-a11b-30ec84f66b67\") " Oct 30 20:56:34.292510 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:34.292458 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c0c1d0-a65b-47a4-a11b-30ec84f66b67-kube-api-access-2csl2\" (UniqueName: \"kubernetes.io/projected/c5c0c1d0-a65b-47a4-a11b-30ec84f66b67-kube-api-access-2csl2\") pod \"c5c0c1d0-a65b-47a4-a11b-30ec84f66b67\" (UID: \"c5c0c1d0-a65b-47a4-a11b-30ec84f66b67\") " Oct 30 20:56:34.292781 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:34.292743 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c0c1d0-a65b-47a4-a11b-30ec84f66b67-plugins-config-volume" pod "c5c0c1d0-a65b-47a4-a11b-30ec84f66b67" (UID: "c5c0c1d0-a65b-47a4-a11b-30ec84f66b67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:56:34.293160 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:34.293138 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c0c1d0-a65b-47a4-a11b-30ec84f66b67-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:34.295370 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:34.295341 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c0c1d0-a65b-47a4-a11b-30ec84f66b67-kube-api-access-2csl2" pod "c5c0c1d0-a65b-47a4-a11b-30ec84f66b67" (UID: "c5c0c1d0-a65b-47a4-a11b-30ec84f66b67"). InnerVolumeSpecName "kube-api-access-2csl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:56:34.396406 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:34.394687 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2csl2\" (UniqueName: \"kubernetes.io/projected/c5c0c1d0-a65b-47a4-a11b-30ec84f66b67-kube-api-access-2csl2\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:35.245242 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:35.245155 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl" event={"ID":"c5c0c1d0-a65b-47a4-a11b-30ec84f66b67","Type":"ContainerDied","Data":"37201b57868ec231d480c398dbc04fc887c229cb4d55a15e1e9cf8dd9a90b30c"} Oct 30 20:56:35.245242 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:35.245232 4848 scope.go:122] "RemoveContainer" containerID="cfa407f945b6d61a00e298a682f009babdfb638c8b46896d9ca932e56038bbca" Oct 30 20:56:35.245649 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:35.245392 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl" Oct 30 20:56:35.278872 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:35.278833 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl"] Oct 30 20:56:35.284721 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:35.284690 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c5bf9a-6b945b7c55-w5zwl"] Oct 30 20:56:36.933708 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:36.933677 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5c0c1d0-a65b-47a4-a11b-30ec84f66b67" path="/var/lib/kubelet/pods/c5c0c1d0-a65b-47a4-a11b-30ec84f66b67/volumes" Oct 30 20:56:37.844388 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:37.844349 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg" podUID="b4d9703d-5037-46a9-ad68-5b5b17618482" containerName="payload-pre-processing" containerID="cri-o://882cc6918378c7a29f8dcab0ce0d6a6806e34143adc078a207465547540754a7" gracePeriod=25 Oct 30 20:56:38.271400 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:38.271370 4848 generic.go:356] "Generic (PLEG): container finished" podID="b4d9703d-5037-46a9-ad68-5b5b17618482" containerID="882cc6918378c7a29f8dcab0ce0d6a6806e34143adc078a207465547540754a7" exitCode=0 Oct 30 20:56:38.271859 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:38.271446 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg" event={"ID":"b4d9703d-5037-46a9-ad68-5b5b17618482","Type":"ContainerDied","Data":"882cc6918378c7a29f8dcab0ce0d6a6806e34143adc078a207465547540754a7"} Oct 30 20:56:38.709557 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:38.709527 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg" Oct 30 20:56:38.806667 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:38.806639 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4d9703d-5037-46a9-ad68-5b5b17618482-kube-api-access-jgl6n\" (UniqueName: \"kubernetes.io/projected/b4d9703d-5037-46a9-ad68-5b5b17618482-kube-api-access-jgl6n\") pod \"b4d9703d-5037-46a9-ad68-5b5b17618482\" (UID: \"b4d9703d-5037-46a9-ad68-5b5b17618482\") " Oct 30 20:56:38.806885 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:38.806825 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4d9703d-5037-46a9-ad68-5b5b17618482-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4d9703d-5037-46a9-ad68-5b5b17618482-plugins-config-volume\") pod \"b4d9703d-5037-46a9-ad68-5b5b17618482\" (UID: \"b4d9703d-5037-46a9-ad68-5b5b17618482\") " Oct 30 20:56:38.807304 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:38.807276 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4d9703d-5037-46a9-ad68-5b5b17618482-plugins-config-volume" pod "b4d9703d-5037-46a9-ad68-5b5b17618482" (UID: "b4d9703d-5037-46a9-ad68-5b5b17618482"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:56:38.807564 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:38.807539 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4d9703d-5037-46a9-ad68-5b5b17618482-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:38.809153 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:38.809127 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d9703d-5037-46a9-ad68-5b5b17618482-kube-api-access-jgl6n" pod "b4d9703d-5037-46a9-ad68-5b5b17618482" (UID: "b4d9703d-5037-46a9-ad68-5b5b17618482"). InnerVolumeSpecName "kube-api-access-jgl6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:56:38.909653 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:38.909622 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgl6n\" (UniqueName: \"kubernetes.io/projected/b4d9703d-5037-46a9-ad68-5b5b17618482-kube-api-access-jgl6n\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:56:39.282260 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:39.282230 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg" Oct 30 20:56:39.282837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:39.282269 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg" event={"ID":"b4d9703d-5037-46a9-ad68-5b5b17618482","Type":"ContainerDied","Data":"d97aef9171b33cdddda81184186ca689f5af1236062f1971b85942d6ce45ee4b"} Oct 30 20:56:39.282837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:39.282353 4848 scope.go:122] "RemoveContainer" containerID="882cc6918378c7a29f8dcab0ce0d6a6806e34143adc078a207465547540754a7" Oct 30 20:56:39.306083 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:39.306055 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg"] Oct 30 20:56:39.314016 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:39.313663 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-c98d5dd55-nn6zg"] Oct 30 20:56:40.291938 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:40.291846 4848 generic.go:356] "Generic (PLEG): container finished" podID="2d0f604c-f9f8-40fb-ab5e-8517520166a3" containerID="8c4aa4c4e8cf753eeceea15dea7b0bc657490cd1de2b382fa629f05006d908b1" exitCode=0 Oct 30 20:56:40.292498 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:40.292064 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" event={"ID":"2d0f604c-f9f8-40fb-ab5e-8517520166a3","Type":"ContainerDied","Data":"8c4aa4c4e8cf753eeceea15dea7b0bc657490cd1de2b382fa629f05006d908b1"} Oct 30 20:56:40.292637 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:40.292616 4848 scope.go:122] "RemoveContainer" containerID="8c4aa4c4e8cf753eeceea15dea7b0bc657490cd1de2b382fa629f05006d908b1" Oct 30 20:56:40.294558 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:40.294527 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:40.294657 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:40.294601 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:40.294657 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:40.294640 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:40.978979 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:40.978895 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4d9703d-5037-46a9-ad68-5b5b17618482" path="/var/lib/kubelet/pods/b4d9703d-5037-46a9-ad68-5b5b17618482/volumes" Oct 30 20:56:41.316538 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.316459 4848 generic.go:356] "Generic (PLEG): container finished" podID="a53b4de2-a49e-44d8-bc2e-369e52309561" containerID="de629f83777a61eb2754c58623a1973ca282be13d91288429c1233a95c42a060" exitCode=0 Oct 30 20:56:41.316538 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.316519 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" event={"ID":"a53b4de2-a49e-44d8-bc2e-369e52309561","Type":"ContainerDied","Data":"de629f83777a61eb2754c58623a1973ca282be13d91288429c1233a95c42a060"} Oct 30 20:56:41.317120 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.317080 4848 scope.go:122] "RemoveContainer" containerID="de629f83777a61eb2754c58623a1973ca282be13d91288429c1233a95c42a060" Oct 30 20:56:41.319019 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.318977 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:41.319150 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.319075 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:41.319150 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.319136 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:56:41.319538 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.319511 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" event={"ID":"2d0f604c-f9f8-40fb-ab5e-8517520166a3","Type":"ContainerStarted","Data":"0535ab83e33041130a6aa473ff5cdaf52048834874a057561e502aff896917ba"} Oct 30 20:56:41.599413 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.599324 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:56:41.600279 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.600252 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 20:56:41.600394 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.600316 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" podUID="2d0f604c-f9f8-40fb-ab5e-8517520166a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 20:56:41.880422 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.880344 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c"] Oct 30 20:56:41.881129 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.881111 4848 cpu_manager.go:414] "containerMap: removing container" podUID="b4d9703d-5037-46a9-ad68-5b5b17618482" containerName="payload-pre-processing" Oct 30 20:56:41.881196 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.881130 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4d9703d-5037-46a9-ad68-5b5b17618482" containerName="payload-pre-processing" Oct 30 20:56:41.881196 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.881159 4848 cpu_manager.go:414] "containerMap: removing container" podUID="c5c0c1d0-a65b-47a4-a11b-30ec84f66b67" containerName="payload-pre-processing" Oct 30 20:56:41.881196 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.881164 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c0c1d0-a65b-47a4-a11b-30ec84f66b67" containerName="payload-pre-processing" Oct 30 20:56:41.881296 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.881282 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4d9703d-5037-46a9-ad68-5b5b17618482" containerName="payload-pre-processing" Oct 30 20:56:41.881296 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.881294 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c0c1d0-a65b-47a4-a11b-30ec84f66b67" containerName="payload-pre-processing" Oct 30 20:56:41.887635 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.887614 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c" Oct 30 20:56:41.889547 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.889522 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c"] Oct 30 20:56:41.894833 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.894155 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a61855\"" Oct 30 20:56:41.898227 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.898204 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx"] Oct 30 20:56:41.903116 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.903093 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx" Oct 30 20:56:41.912789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.912638 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a61855-dockercfg-4d7bd\"" Oct 30 20:56:41.952159 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:41.952122 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx"] Oct 30 20:56:42.088791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:42.087912 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcs2m\" (UniqueName: \"kubernetes.io/projected/0d25fd62-4190-4453-bf50-38d631d53cf1-kube-api-access-fcs2m\") pod \"payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c\" (UID: \"0d25fd62-4190-4453-bf50-38d631d53cf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c" Oct 30 20:56:42.088791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:42.088011 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d25fd62-4190-4453-bf50-38d631d53cf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c\" (UID: \"0d25fd62-4190-4453-bf50-38d631d53cf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c" Oct 30 20:56:42.088791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:42.088111 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvn2z\" (UniqueName: \"kubernetes.io/projected/9ca3d842-f891-4981-a687-6971c181b6b7-kube-api-access-rvn2z\") pod \"payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx\" (UID: \"9ca3d842-f891-4981-a687-6971c181b6b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx" Oct 30 20:56:42.088791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:42.088246 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca3d842-f891-4981-a687-6971c181b6b7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx\" (UID: \"9ca3d842-f891-4981-a687-6971c181b6b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx" Oct 30 20:56:42.199803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:42.191869 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcs2m\" (UniqueName: \"kubernetes.io/projected/0d25fd62-4190-4453-bf50-38d631d53cf1-kube-api-access-fcs2m\") pod \"payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c\" (UID: \"0d25fd62-4190-4453-bf50-38d631d53cf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c" Oct 30 20:56:42.199803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:42.191965 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d25fd62-4190-4453-bf50-38d631d53cf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c\" (UID: \"0d25fd62-4190-4453-bf50-38d631d53cf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c" Oct 30 20:56:42.199803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:42.192034 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvn2z\" (UniqueName: \"kubernetes.io/projected/9ca3d842-f891-4981-a687-6971c181b6b7-kube-api-access-rvn2z\") pod \"payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx\" (UID: \"9ca3d842-f891-4981-a687-6971c181b6b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx" Oct 30 20:56:42.199803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:42.192151 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca3d842-f891-4981-a687-6971c181b6b7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx\" (UID: \"9ca3d842-f891-4981-a687-6971c181b6b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx" Oct 30 20:56:42.199803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:42.193174 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca3d842-f891-4981-a687-6971c181b6b7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx\" (UID: \"9ca3d842-f891-4981-a687-6971c181b6b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx" Oct 30 20:56:42.199803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:42.194329 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d25fd62-4190-4453-bf50-38d631d53cf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c\" (UID: \"0d25fd62-4190-4453-bf50-38d631d53cf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c" Oct 30 20:56:42.232788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:42.232547 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvn2z\" (UniqueName: \"kubernetes.io/projected/9ca3d842-f891-4981-a687-6971c181b6b7-kube-api-access-rvn2z\") pod \"payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx\" (UID: \"9ca3d842-f891-4981-a687-6971c181b6b7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx" Oct 30 20:56:42.243743 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:42.243711 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcs2m\" (UniqueName: \"kubernetes.io/projected/0d25fd62-4190-4453-bf50-38d631d53cf1-kube-api-access-fcs2m\") pod \"payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c\" (UID: \"0d25fd62-4190-4453-bf50-38d631d53cf1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c" Oct 30 20:56:42.271531 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:42.269048 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c" Oct 30 20:56:42.290186 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:42.284101 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx" Oct 30 20:56:42.486639 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:42.484446 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" event={"ID":"a53b4de2-a49e-44d8-bc2e-369e52309561","Type":"ContainerStarted","Data":"4a3c6f0f4c15e09a78281bd86a8dedaf4851c8ca9058caa6087e3370304aeb12"} Oct 30 20:56:42.619016 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:42.615481 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 20:56:42.619016 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:42.615540 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" podUID="2d0f604c-f9f8-40fb-ab5e-8517520166a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 20:56:42.727941 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:42.702341 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:56:42.727941 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:42.702454 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 20:56:42.727941 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:42.702493 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" podUID="a53b4de2-a49e-44d8-bc2e-369e52309561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 20:56:43.266889 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:43.266690 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx"] Oct 30 20:56:43.286969 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:43.286938 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c"] Oct 30 20:56:43.286969 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:56:43.286934 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d25fd62_4190_4453_bf50_38d631d53cf1.slice/crio-da2e27c7c917a981f906502365c96a8acaa4bc64ca8e714c9b1db1918b2afaea WatchSource:0}: Error finding container da2e27c7c917a981f906502365c96a8acaa4bc64ca8e714c9b1db1918b2afaea: Status 404 returned error can't find the container with id da2e27c7c917a981f906502365c96a8acaa4bc64ca8e714c9b1db1918b2afaea Oct 30 20:56:43.486666 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:43.486619 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx" event={"ID":"9ca3d842-f891-4981-a687-6971c181b6b7","Type":"ContainerStarted","Data":"b6a1a662f3bee82f46502f2bf21c4206e02b2703ea89ab520f39bba5599892cd"} Oct 30 20:56:43.491799 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:43.489191 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c" event={"ID":"0d25fd62-4190-4453-bf50-38d631d53cf1","Type":"ContainerStarted","Data":"da2e27c7c917a981f906502365c96a8acaa4bc64ca8e714c9b1db1918b2afaea"} Oct 30 20:56:43.601187 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:43.601147 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 20:56:43.601418 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:43.601208 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" podUID="2d0f604c-f9f8-40fb-ab5e-8517520166a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 20:56:43.695850 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:43.695062 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 20:56:43.695850 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:43.695149 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" podUID="a53b4de2-a49e-44d8-bc2e-369e52309561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 20:56:44.500908 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:44.500564 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c" event={"ID":"0d25fd62-4190-4453-bf50-38d631d53cf1","Type":"ContainerStarted","Data":"dd4360eeaf2d216c9e993a3287dfaeb85de6faffaa9678ba77035170a0c34288"} Oct 30 20:56:44.501397 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:44.500979 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c" Oct 30 20:56:44.512799 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:44.503500 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx" event={"ID":"9ca3d842-f891-4981-a687-6971c181b6b7","Type":"ContainerStarted","Data":"383fe338c7397ef533b04e645bc075e37c5b3475823f1dc3d6fffe9023eedb98"} Oct 30 20:56:44.512799 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:44.504331 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx" Oct 30 20:56:44.564631 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:44.564563 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c" podStartSLOduration=3.564544132 podStartE2EDuration="3.564544132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:44.529880287 +0000 UTC m=+1109.709776796" watchObservedRunningTime="2026-10-30 20:56:44.564544132 +0000 UTC m=+1109.744440640" Oct 30 20:56:44.564855 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:44.564688 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx" podStartSLOduration=3.5646795129999997 podStartE2EDuration="3.564679513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:56:44.561188443 +0000 UTC m=+1109.741084967" watchObservedRunningTime="2026-10-30 20:56:44.564679513 +0000 UTC m=+1109.744576024" Oct 30 20:56:44.602208 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:44.602175 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 20:56:44.602507 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:44.602469 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" podUID="2d0f604c-f9f8-40fb-ab5e-8517520166a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 20:56:44.698804 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:44.698514 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 20:56:44.698804 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:44.698585 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" podUID="a53b4de2-a49e-44d8-bc2e-369e52309561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 20:56:45.617272 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:45.602992 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 20:56:45.617272 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:45.603052 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" podUID="2d0f604c-f9f8-40fb-ab5e-8517520166a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 20:56:45.710791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:45.710445 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 20:56:45.710791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:45.710519 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" podUID="a53b4de2-a49e-44d8-bc2e-369e52309561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 20:56:46.599281 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:46.599251 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:56:46.599549 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:46.599537 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:56:46.599798 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:46.599776 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 20:56:46.600553 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:46.600531 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" podUID="2d0f604c-f9f8-40fb-ab5e-8517520166a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 20:56:46.717803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:46.713975 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 20:56:46.717803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:46.714040 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" podUID="a53b4de2-a49e-44d8-bc2e-369e52309561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 20:56:47.601789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:47.601446 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 20:56:47.601789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:47.601511 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" podUID="2d0f604c-f9f8-40fb-ab5e-8517520166a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 20:56:47.694455 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:47.694410 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 20:56:47.694645 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:47.694469 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" podUID="a53b4de2-a49e-44d8-bc2e-369e52309561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 20:56:48.600229 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:48.600194 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 20:56:48.600877 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:48.600825 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" podUID="2d0f604c-f9f8-40fb-ab5e-8517520166a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 20:56:48.695101 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:48.695065 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 20:56:48.695356 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:48.695323 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" podUID="a53b4de2-a49e-44d8-bc2e-369e52309561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 20:56:49.699790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:49.697190 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 20:56:49.699790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:49.697252 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" podUID="a53b4de2-a49e-44d8-bc2e-369e52309561" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 20:56:49.854683 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:49.854620 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:56:49.861707 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:49.861677 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:56:50.733187 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:50.733150 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:56:50.747680 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:50.747650 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:56:52.540713 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:56:52.540672 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:56:56.007819 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:56.007788 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c" Oct 30 20:56:57.065435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:56:57.065398 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx" Oct 30 20:57:00.342897 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:00.342789 4848 generic.go:356] "Generic (PLEG): container finished" podID="0c4e64ec-eb49-4a9e-86df-7483d597fd9c" containerID="95b6dc03d99fc0248e0083fa8cd12911f66e177064eb0980fd6b9ba309ba5fcc" exitCode=0 Oct 30 20:57:00.343634 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:00.342903 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" event={"ID":"0c4e64ec-eb49-4a9e-86df-7483d597fd9c","Type":"ContainerDied","Data":"95b6dc03d99fc0248e0083fa8cd12911f66e177064eb0980fd6b9ba309ba5fcc"} Oct 30 20:57:00.343634 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:00.343557 4848 scope.go:122] "RemoveContainer" containerID="95b6dc03d99fc0248e0083fa8cd12911f66e177064eb0980fd6b9ba309ba5fcc" Oct 30 20:57:00.351520 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:00.351474 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:00.351675 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:00.351576 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:00.351675 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:00.351626 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:00.399048 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:00.399013 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:57:01.406789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:01.403371 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" event={"ID":"0c4e64ec-eb49-4a9e-86df-7483d597fd9c","Type":"ContainerStarted","Data":"f3a46c24f7495f934a072dd252625430b6bde5607733410e883217c324c7e64f"} Oct 30 20:57:01.927789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:01.927230 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 20:57:01.927789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:01.927317 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" podUID="0c4e64ec-eb49-4a9e-86df-7483d597fd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 20:57:01.927789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:01.927721 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:57:02.932789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:02.928089 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 20:57:02.932789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:02.928890 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" podUID="0c4e64ec-eb49-4a9e-86df-7483d597fd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 20:57:03.938792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:03.927317 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 20:57:03.938792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:03.927392 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" podUID="0c4e64ec-eb49-4a9e-86df-7483d597fd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 20:57:04.971574 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:04.971540 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 20:57:04.972099 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:04.971979 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" podUID="0c4e64ec-eb49-4a9e-86df-7483d597fd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 20:57:05.925819 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:05.924173 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 20:57:05.925819 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:05.924239 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" podUID="0c4e64ec-eb49-4a9e-86df-7483d597fd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 20:57:06.950690 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:06.950644 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 20:57:06.951354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:06.951298 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" podUID="0c4e64ec-eb49-4a9e-86df-7483d597fd9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 20:57:08.074308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:08.074283 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:57:08.075652 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:08.075632 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:57:08.079037 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:08.079015 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:57:09.217281 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:09.217249 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk"] Oct 30 20:57:09.218681 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:09.218633 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" podUID="07654e83-e9e1-4297-be2b-41591a932b39" containerName="maas-api" containerID="cri-o://db59858308e17190b388ace19ca6b8414b1b9594369d12df322d9df456408080" gracePeriod=30 Oct 30 20:57:09.855380 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:09.854088 4848 generic.go:356] "Generic (PLEG): container finished" podID="07654e83-e9e1-4297-be2b-41591a932b39" containerID="db59858308e17190b388ace19ca6b8414b1b9594369d12df322d9df456408080" exitCode=0 Oct 30 20:57:09.855380 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:09.855313 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" event={"ID":"07654e83-e9e1-4297-be2b-41591a932b39","Type":"ContainerDied","Data":"db59858308e17190b388ace19ca6b8414b1b9594369d12df322d9df456408080"} Oct 30 20:57:09.903254 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:09.902903 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" Oct 30 20:57:10.006790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:09.999095 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07654e83-e9e1-4297-be2b-41591a932b39-metrics-certs\" (UniqueName: \"kubernetes.io/secret/07654e83-e9e1-4297-be2b-41591a932b39-metrics-certs\") pod \"07654e83-e9e1-4297-be2b-41591a932b39\" (UID: \"07654e83-e9e1-4297-be2b-41591a932b39\") " Oct 30 20:57:10.006790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:09.999178 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07654e83-e9e1-4297-be2b-41591a932b39-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07654e83-e9e1-4297-be2b-41591a932b39-maas-api-tls\") pod \"07654e83-e9e1-4297-be2b-41591a932b39\" (UID: \"07654e83-e9e1-4297-be2b-41591a932b39\") " Oct 30 20:57:10.006790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:09.999512 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07654e83-e9e1-4297-be2b-41591a932b39-kube-api-access-hxw8q\" (UniqueName: \"kubernetes.io/projected/07654e83-e9e1-4297-be2b-41591a932b39-kube-api-access-hxw8q\") pod \"07654e83-e9e1-4297-be2b-41591a932b39\" (UID: \"07654e83-e9e1-4297-be2b-41591a932b39\") " Oct 30 20:57:10.006790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:09.999636 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07654e83-e9e1-4297-be2b-41591a932b39-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07654e83-e9e1-4297-be2b-41591a932b39-openshift-service-ca\") pod \"07654e83-e9e1-4297-be2b-41591a932b39\" (UID: \"07654e83-e9e1-4297-be2b-41591a932b39\") " Oct 30 20:57:10.006790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:10.001566 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07654e83-e9e1-4297-be2b-41591a932b39-openshift-service-ca" pod "07654e83-e9e1-4297-be2b-41591a932b39" (UID: "07654e83-e9e1-4297-be2b-41591a932b39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:10.021834 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:10.012017 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07654e83-e9e1-4297-be2b-41591a932b39-kube-api-access-hxw8q" pod "07654e83-e9e1-4297-be2b-41591a932b39" (UID: "07654e83-e9e1-4297-be2b-41591a932b39"). InnerVolumeSpecName "kube-api-access-hxw8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:10.021993 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:10.021965 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07654e83-e9e1-4297-be2b-41591a932b39-metrics-certs" pod "07654e83-e9e1-4297-be2b-41591a932b39" (UID: "07654e83-e9e1-4297-be2b-41591a932b39"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:10.035959 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:10.035921 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07654e83-e9e1-4297-be2b-41591a932b39-maas-api-tls" pod "07654e83-e9e1-4297-be2b-41591a932b39" (UID: "07654e83-e9e1-4297-be2b-41591a932b39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:10.127147 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:10.106504 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxw8q\" (UniqueName: \"kubernetes.io/projected/07654e83-e9e1-4297-be2b-41591a932b39-kube-api-access-hxw8q\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:10.127147 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:10.106541 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07654e83-e9e1-4297-be2b-41591a932b39-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:10.127147 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:10.106556 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07654e83-e9e1-4297-be2b-41591a932b39-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:10.127147 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:10.106569 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07654e83-e9e1-4297-be2b-41591a932b39-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:10.865785 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:10.864688 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" event={"ID":"07654e83-e9e1-4297-be2b-41591a932b39","Type":"ContainerDied","Data":"20951f7ee1140051b17e20a9595b1b66b12606a35bdd0192bb46835a055f13e8"} Oct 30 20:57:10.865785 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:10.864799 4848 scope.go:122] "RemoveContainer" containerID="db59858308e17190b388ace19ca6b8414b1b9594369d12df322d9df456408080" Oct 30 20:57:10.865785 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:10.864709 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk" Oct 30 20:57:10.899653 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:10.899619 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk"] Oct 30 20:57:10.907984 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:10.907941 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5e71c9-8d64cbbb8-gf7nk"] Oct 30 20:57:10.934736 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:10.934692 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07654e83-e9e1-4297-be2b-41591a932b39" path="/var/lib/kubelet/pods/07654e83-e9e1-4297-be2b-41591a932b39/volumes" Oct 30 20:57:13.956906 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:13.956873 4848 generic.go:356] "Generic (PLEG): container finished" podID="3f3e6ed8-9d1e-4716-ab89-a37141dc030d" containerID="29be838c4e202a929f01d3ae03df0128512314e8de2d5fd95bea31235a40a046" exitCode=0 Oct 30 20:57:13.957461 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:13.956951 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" event={"ID":"3f3e6ed8-9d1e-4716-ab89-a37141dc030d","Type":"ContainerDied","Data":"29be838c4e202a929f01d3ae03df0128512314e8de2d5fd95bea31235a40a046"} Oct 30 20:57:13.957552 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:13.957533 4848 scope.go:122] "RemoveContainer" containerID="29be838c4e202a929f01d3ae03df0128512314e8de2d5fd95bea31235a40a046" Oct 30 20:57:13.959392 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:13.959360 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:13.959488 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:13.959433 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:13.959488 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:13.959462 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:14.965960 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:14.965905 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" event={"ID":"3f3e6ed8-9d1e-4716-ab89-a37141dc030d","Type":"ContainerStarted","Data":"c24557ce0d91c6c7907a6357bf8e52025fa5ff9da43cf1350d811d00167c5c17"} Oct 30 20:57:15.153281 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:15.153246 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:57:15.377384 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:15.377297 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:57:15.385835 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:15.385798 4848 patch_prober.go:29] interesting pod/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 20:57:15.386122 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:15.386091 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" podUID="3f3e6ed8-9d1e-4716-ab89-a37141dc030d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 20:57:16.373453 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:16.373353 4848 patch_prober.go:29] interesting pod/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 20:57:16.373453 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:16.373439 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" podUID="3f3e6ed8-9d1e-4716-ab89-a37141dc030d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 20:57:17.396800 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:17.374152 4848 patch_prober.go:29] interesting pod/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 20:57:17.397461 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:17.397409 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" podUID="3f3e6ed8-9d1e-4716-ab89-a37141dc030d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 20:57:17.700256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:17.700207 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2"] Oct 30 20:57:18.423271 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:18.423106 4848 patch_prober.go:29] interesting pod/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 20:57:18.423271 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:18.423178 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" podUID="3f3e6ed8-9d1e-4716-ab89-a37141dc030d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 20:57:18.530078 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:18.530043 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw"] Oct 30 20:57:18.530577 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:18.530541 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" podUID="bf9525d3-a53f-4e30-8fae-12a407eace4e" containerName="istio-proxy" containerID="cri-o://b5e2414fd2c098487de261474686523438e6a43c75b4887e369f2ebf909d5bd3" gracePeriod=30 Oct 30 20:57:19.154231 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:19.154171 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:57:19.154580 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:19.154519 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" podUID="bf9525d3-a53f-4e30-8fae-12a407eace4e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:57:19.374679 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:19.374639 4848 patch_prober.go:29] interesting pod/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 20:57:19.374853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:19.374698 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" podUID="3f3e6ed8-9d1e-4716-ab89-a37141dc030d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 20:57:20.373445 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:20.373410 4848 patch_prober.go:29] interesting pod/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 30 20:57:20.373976 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:20.373478 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" podUID="3f3e6ed8-9d1e-4716-ab89-a37141dc030d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 30 20:57:21.531802 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:21.530365 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:57:21.539837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:21.537985 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:57:22.160668 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:22.160635 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh"] Oct 30 20:57:22.161189 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:22.161158 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" podUID="849dd3a0-afee-4409-be49-d0a264b2b31b" containerName="maas-api" containerID="cri-o://5882201326f2cdb8865f807b883d2c7badff866e09ec5661edfc43d5e5615529" gracePeriod=30 Oct 30 20:57:22.716818 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:22.710227 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2" podUID="9f9f3012-8124-49be-92c4-3f04feb2a82c" containerName="payload-pre-processing" containerID="cri-o://98738c47a510fcd947330e9cd6dec89397e68a1bbdf0cd00473e65156866a153" gracePeriod=25 Oct 30 20:57:22.809840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:22.783760 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" Oct 30 20:57:22.828888 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:57:22.817006 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:57:22.875852 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:22.856289 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/849dd3a0-afee-4409-be49-d0a264b2b31b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/849dd3a0-afee-4409-be49-d0a264b2b31b-openshift-service-ca\") pod \"849dd3a0-afee-4409-be49-d0a264b2b31b\" (UID: \"849dd3a0-afee-4409-be49-d0a264b2b31b\") " Oct 30 20:57:22.875852 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:22.856434 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/849dd3a0-afee-4409-be49-d0a264b2b31b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/849dd3a0-afee-4409-be49-d0a264b2b31b-maas-api-tls\") pod \"849dd3a0-afee-4409-be49-d0a264b2b31b\" (UID: \"849dd3a0-afee-4409-be49-d0a264b2b31b\") " Oct 30 20:57:22.875852 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:22.856536 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/849dd3a0-afee-4409-be49-d0a264b2b31b-kube-api-access-p45m9\" (UniqueName: \"kubernetes.io/projected/849dd3a0-afee-4409-be49-d0a264b2b31b-kube-api-access-p45m9\") pod \"849dd3a0-afee-4409-be49-d0a264b2b31b\" (UID: \"849dd3a0-afee-4409-be49-d0a264b2b31b\") " Oct 30 20:57:22.875852 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:22.856677 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/849dd3a0-afee-4409-be49-d0a264b2b31b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/849dd3a0-afee-4409-be49-d0a264b2b31b-metrics-certs\") pod \"849dd3a0-afee-4409-be49-d0a264b2b31b\" (UID: \"849dd3a0-afee-4409-be49-d0a264b2b31b\") " Oct 30 20:57:22.875852 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:22.858680 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/849dd3a0-afee-4409-be49-d0a264b2b31b-openshift-service-ca" pod "849dd3a0-afee-4409-be49-d0a264b2b31b" (UID: "849dd3a0-afee-4409-be49-d0a264b2b31b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:22.906556 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:22.891094 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/849dd3a0-afee-4409-be49-d0a264b2b31b-maas-api-tls" pod "849dd3a0-afee-4409-be49-d0a264b2b31b" (UID: "849dd3a0-afee-4409-be49-d0a264b2b31b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:22.919786 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:22.919532 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/849dd3a0-afee-4409-be49-d0a264b2b31b-metrics-certs" pod "849dd3a0-afee-4409-be49-d0a264b2b31b" (UID: "849dd3a0-afee-4409-be49-d0a264b2b31b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:22.919786 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:22.919683 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849dd3a0-afee-4409-be49-d0a264b2b31b-kube-api-access-p45m9" pod "849dd3a0-afee-4409-be49-d0a264b2b31b" (UID: "849dd3a0-afee-4409-be49-d0a264b2b31b"). InnerVolumeSpecName "kube-api-access-p45m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:22.966147 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:22.965796 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/849dd3a0-afee-4409-be49-d0a264b2b31b-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:22.966147 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:22.965826 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/849dd3a0-afee-4409-be49-d0a264b2b31b-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:22.966147 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:22.965839 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p45m9\" (UniqueName: \"kubernetes.io/projected/849dd3a0-afee-4409-be49-d0a264b2b31b-kube-api-access-p45m9\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:22.966147 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:22.965853 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/849dd3a0-afee-4409-be49-d0a264b2b31b-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:23.177422 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:23.177385 4848 generic.go:356] "Generic (PLEG): container finished" podID="849dd3a0-afee-4409-be49-d0a264b2b31b" containerID="5882201326f2cdb8865f807b883d2c7badff866e09ec5661edfc43d5e5615529" exitCode=0 Oct 30 20:57:23.178668 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:23.177734 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" Oct 30 20:57:23.178668 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:23.177751 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" event={"ID":"849dd3a0-afee-4409-be49-d0a264b2b31b","Type":"ContainerDied","Data":"5882201326f2cdb8865f807b883d2c7badff866e09ec5661edfc43d5e5615529"} Oct 30 20:57:23.178668 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:23.177858 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh" event={"ID":"849dd3a0-afee-4409-be49-d0a264b2b31b","Type":"ContainerDied","Data":"3df91fa4f683842525d3cb7855c025bdda8019afa04cb75fcb03402fd65d4499"} Oct 30 20:57:23.178668 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:23.177890 4848 scope.go:122] "RemoveContainer" containerID="5882201326f2cdb8865f807b883d2c7badff866e09ec5661edfc43d5e5615529" Oct 30 20:57:23.188020 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:23.187959 4848 generic.go:356] "Generic (PLEG): container finished" podID="9f9f3012-8124-49be-92c4-3f04feb2a82c" containerID="98738c47a510fcd947330e9cd6dec89397e68a1bbdf0cd00473e65156866a153" exitCode=0 Oct 30 20:57:23.188387 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:23.188296 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2" event={"ID":"9f9f3012-8124-49be-92c4-3f04feb2a82c","Type":"ContainerDied","Data":"98738c47a510fcd947330e9cd6dec89397e68a1bbdf0cd00473e65156866a153"} Oct 30 20:57:23.233553 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:23.233525 4848 scope.go:122] "RemoveContainer" containerID="5882201326f2cdb8865f807b883d2c7badff866e09ec5661edfc43d5e5615529" Oct 30 20:57:23.239694 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:57:23.239650 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5882201326f2cdb8865f807b883d2c7badff866e09ec5661edfc43d5e5615529\": container with ID starting with 5882201326f2cdb8865f807b883d2c7badff866e09ec5661edfc43d5e5615529 not found: ID does not exist" containerID="5882201326f2cdb8865f807b883d2c7badff866e09ec5661edfc43d5e5615529" Oct 30 20:57:23.239959 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:23.239927 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5882201326f2cdb8865f807b883d2c7badff866e09ec5661edfc43d5e5615529"} err="failed to get container status \"5882201326f2cdb8865f807b883d2c7badff866e09ec5661edfc43d5e5615529\": rpc error: code = NotFound desc = could not find container \"5882201326f2cdb8865f807b883d2c7badff866e09ec5661edfc43d5e5615529\": container with ID starting with 5882201326f2cdb8865f807b883d2c7badff866e09ec5661edfc43d5e5615529 not found: ID does not exist" Oct 30 20:57:23.255408 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:23.255354 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh"] Oct 30 20:57:23.261818 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:23.261788 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-eebcb4-56cdd44845-jzdqh"] Oct 30 20:57:23.585064 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:23.585032 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2" Oct 30 20:57:23.739949 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:23.739921 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f9f3012-8124-49be-92c4-3f04feb2a82c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f9f3012-8124-49be-92c4-3f04feb2a82c-plugins-config-volume\") pod \"9f9f3012-8124-49be-92c4-3f04feb2a82c\" (UID: \"9f9f3012-8124-49be-92c4-3f04feb2a82c\") " Oct 30 20:57:23.740344 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:23.740081 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f9f3012-8124-49be-92c4-3f04feb2a82c-kube-api-access-5t56f\" (UniqueName: \"kubernetes.io/projected/9f9f3012-8124-49be-92c4-3f04feb2a82c-kube-api-access-5t56f\") pod \"9f9f3012-8124-49be-92c4-3f04feb2a82c\" (UID: \"9f9f3012-8124-49be-92c4-3f04feb2a82c\") " Oct 30 20:57:23.740400 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:23.740338 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f9f3012-8124-49be-92c4-3f04feb2a82c-plugins-config-volume" pod "9f9f3012-8124-49be-92c4-3f04feb2a82c" (UID: "9f9f3012-8124-49be-92c4-3f04feb2a82c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:23.740560 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:23.740542 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f9f3012-8124-49be-92c4-3f04feb2a82c-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:23.742184 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:23.742161 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f9f3012-8124-49be-92c4-3f04feb2a82c-kube-api-access-5t56f" pod "9f9f3012-8124-49be-92c4-3f04feb2a82c" (UID: "9f9f3012-8124-49be-92c4-3f04feb2a82c"). InnerVolumeSpecName "kube-api-access-5t56f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:23.842208 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:23.842115 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t56f\" (UniqueName: \"kubernetes.io/projected/9f9f3012-8124-49be-92c4-3f04feb2a82c-kube-api-access-5t56f\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:24.041783 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.041748 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:57:24.147037 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.146957 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf9525d3-a53f-4e30-8fae-12a407eace4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9525d3-a53f-4e30-8fae-12a407eace4e-istiod-ca-cert\") pod \"bf9525d3-a53f-4e30-8fae-12a407eace4e\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " Oct 30 20:57:24.147037 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.146991 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-podinfo\") pod \"bf9525d3-a53f-4e30-8fae-12a407eace4e\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " Oct 30 20:57:24.147273 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.147069 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-data\") pod \"bf9525d3-a53f-4e30-8fae-12a407eace4e\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " Oct 30 20:57:24.147273 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.147101 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-workload-socket\") pod \"bf9525d3-a53f-4e30-8fae-12a407eace4e\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " Oct 30 20:57:24.147389 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.147283 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-envoy\") pod \"bf9525d3-a53f-4e30-8fae-12a407eace4e\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " Oct 30 20:57:24.147389 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.147306 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-token\" (UniqueName: \"kubernetes.io/projected/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-token\") pod \"bf9525d3-a53f-4e30-8fae-12a407eace4e\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " Oct 30 20:57:24.147389 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.147342 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-workload-certs\") pod \"bf9525d3-a53f-4e30-8fae-12a407eace4e\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " Oct 30 20:57:24.147389 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.147361 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-credential-socket\") pod \"bf9525d3-a53f-4e30-8fae-12a407eace4e\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " Oct 30 20:57:24.147672 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.147411 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf9525d3-a53f-4e30-8fae-12a407eace4e-kube-api-access-jl7x4\" (UniqueName: \"kubernetes.io/projected/bf9525d3-a53f-4e30-8fae-12a407eace4e-kube-api-access-jl7x4\") pod \"bf9525d3-a53f-4e30-8fae-12a407eace4e\" (UID: \"bf9525d3-a53f-4e30-8fae-12a407eace4e\") " Oct 30 20:57:24.147672 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.147409 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf9525d3-a53f-4e30-8fae-12a407eace4e-istiod-ca-cert" pod "bf9525d3-a53f-4e30-8fae-12a407eace4e" (UID: "bf9525d3-a53f-4e30-8fae-12a407eace4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:24.147796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.147687 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-data" pod "bf9525d3-a53f-4e30-8fae-12a407eace4e" (UID: "bf9525d3-a53f-4e30-8fae-12a407eace4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:24.147903 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.147877 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-workload-socket" pod "bf9525d3-a53f-4e30-8fae-12a407eace4e" (UID: "bf9525d3-a53f-4e30-8fae-12a407eace4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:24.148161 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.148142 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf9525d3-a53f-4e30-8fae-12a407eace4e-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:24.148228 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.148166 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:24.148228 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.148182 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:24.148650 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.148608 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-credential-socket" pod "bf9525d3-a53f-4e30-8fae-12a407eace4e" (UID: "bf9525d3-a53f-4e30-8fae-12a407eace4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:24.148759 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.148669 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-workload-certs" pod "bf9525d3-a53f-4e30-8fae-12a407eace4e" (UID: "bf9525d3-a53f-4e30-8fae-12a407eace4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:24.149488 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.149461 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-podinfo" pod "bf9525d3-a53f-4e30-8fae-12a407eace4e" (UID: "bf9525d3-a53f-4e30-8fae-12a407eace4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:57:24.150837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.150813 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9525d3-a53f-4e30-8fae-12a407eace4e-kube-api-access-jl7x4" pod "bf9525d3-a53f-4e30-8fae-12a407eace4e" (UID: "bf9525d3-a53f-4e30-8fae-12a407eace4e"). InnerVolumeSpecName "kube-api-access-jl7x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:24.150925 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.150861 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-token" pod "bf9525d3-a53f-4e30-8fae-12a407eace4e" (UID: "bf9525d3-a53f-4e30-8fae-12a407eace4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:24.150925 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.150862 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-envoy" pod "bf9525d3-a53f-4e30-8fae-12a407eace4e" (UID: "bf9525d3-a53f-4e30-8fae-12a407eace4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:24.196206 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.196156 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2" event={"ID":"9f9f3012-8124-49be-92c4-3f04feb2a82c","Type":"ContainerDied","Data":"8fb9bdad35fa2de029c41ca719725ec1746d808a6b7d6b634ccb8b6a191b3b11"} Oct 30 20:57:24.196206 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.196197 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2" Oct 30 20:57:24.196431 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.196230 4848 scope.go:122] "RemoveContainer" containerID="98738c47a510fcd947330e9cd6dec89397e68a1bbdf0cd00473e65156866a153" Oct 30 20:57:24.228792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.227157 4848 generic.go:356] "Generic (PLEG): container finished" podID="bf9525d3-a53f-4e30-8fae-12a407eace4e" containerID="b5e2414fd2c098487de261474686523438e6a43c75b4887e369f2ebf909d5bd3" exitCode=0 Oct 30 20:57:24.228792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.227353 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" event={"ID":"bf9525d3-a53f-4e30-8fae-12a407eace4e","Type":"ContainerDied","Data":"b5e2414fd2c098487de261474686523438e6a43c75b4887e369f2ebf909d5bd3"} Oct 30 20:57:24.228792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.227407 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" event={"ID":"bf9525d3-a53f-4e30-8fae-12a407eace4e","Type":"ContainerDied","Data":"d0b3278513c2fc1b03cffd2f6149c4e126abf4f258470692cd0df4caa3a04a27"} Oct 30 20:57:24.228792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.227496 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw" Oct 30 20:57:24.239541 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.239516 4848 scope.go:122] "RemoveContainer" containerID="b5e2414fd2c098487de261474686523438e6a43c75b4887e369f2ebf909d5bd3" Oct 30 20:57:24.249454 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.249430 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:24.249454 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.249454 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:24.249585 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.249464 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:24.249585 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.249472 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf9525d3-a53f-4e30-8fae-12a407eace4e-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:24.249585 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.249482 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl7x4\" (UniqueName: \"kubernetes.io/projected/bf9525d3-a53f-4e30-8fae-12a407eace4e-kube-api-access-jl7x4\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:24.249585 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.249491 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf9525d3-a53f-4e30-8fae-12a407eace4e-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:24.249806 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.249717 4848 scope.go:122] "RemoveContainer" containerID="1fe0afeb0b041b6f6c051e19cebc0b21d1e239f02ac85013d8ede4f6396dfe74" Oct 30 20:57:24.260477 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.260447 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2"] Oct 30 20:57:24.261282 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.261259 4848 scope.go:122] "RemoveContainer" containerID="b5e2414fd2c098487de261474686523438e6a43c75b4887e369f2ebf909d5bd3" Oct 30 20:57:24.261901 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:57:24.261872 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e2414fd2c098487de261474686523438e6a43c75b4887e369f2ebf909d5bd3\": container with ID starting with b5e2414fd2c098487de261474686523438e6a43c75b4887e369f2ebf909d5bd3 not found: ID does not exist" containerID="b5e2414fd2c098487de261474686523438e6a43c75b4887e369f2ebf909d5bd3" Oct 30 20:57:24.262010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.261912 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e2414fd2c098487de261474686523438e6a43c75b4887e369f2ebf909d5bd3"} err="failed to get container status \"b5e2414fd2c098487de261474686523438e6a43c75b4887e369f2ebf909d5bd3\": rpc error: code = NotFound desc = could not find container \"b5e2414fd2c098487de261474686523438e6a43c75b4887e369f2ebf909d5bd3\": container with ID starting with b5e2414fd2c098487de261474686523438e6a43c75b4887e369f2ebf909d5bd3 not found: ID does not exist" Oct 30 20:57:24.262010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.261937 4848 scope.go:122] "RemoveContainer" containerID="1fe0afeb0b041b6f6c051e19cebc0b21d1e239f02ac85013d8ede4f6396dfe74" Oct 30 20:57:24.262306 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:57:24.262276 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe0afeb0b041b6f6c051e19cebc0b21d1e239f02ac85013d8ede4f6396dfe74\": container with ID starting with 1fe0afeb0b041b6f6c051e19cebc0b21d1e239f02ac85013d8ede4f6396dfe74 not found: ID does not exist" containerID="1fe0afeb0b041b6f6c051e19cebc0b21d1e239f02ac85013d8ede4f6396dfe74" Oct 30 20:57:24.262399 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.262315 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe0afeb0b041b6f6c051e19cebc0b21d1e239f02ac85013d8ede4f6396dfe74"} err="failed to get container status \"1fe0afeb0b041b6f6c051e19cebc0b21d1e239f02ac85013d8ede4f6396dfe74\": rpc error: code = NotFound desc = could not find container \"1fe0afeb0b041b6f6c051e19cebc0b21d1e239f02ac85013d8ede4f6396dfe74\": container with ID starting with 1fe0afeb0b041b6f6c051e19cebc0b21d1e239f02ac85013d8ede4f6396dfe74 not found: ID does not exist" Oct 30 20:57:24.265060 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.265037 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-319188-5db4ddbd87-hprl2"] Oct 30 20:57:24.274992 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.274917 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw"] Oct 30 20:57:24.278283 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.278262 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5e71c9-openshift-default-6b4bdcb695-n98bw"] Oct 30 20:57:24.935066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.935032 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="849dd3a0-afee-4409-be49-d0a264b2b31b" path="/var/lib/kubelet/pods/849dd3a0-afee-4409-be49-d0a264b2b31b/volumes" Oct 30 20:57:24.935678 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.935660 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f9f3012-8124-49be-92c4-3f04feb2a82c" path="/var/lib/kubelet/pods/9f9f3012-8124-49be-92c4-3f04feb2a82c/volumes" Oct 30 20:57:24.936337 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:24.936320 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf9525d3-a53f-4e30-8fae-12a407eace4e" path="/var/lib/kubelet/pods/bf9525d3-a53f-4e30-8fae-12a407eace4e/volumes" Oct 30 20:57:28.560987 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:28.560942 4848 patch_prober.go:29] interesting pod/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:57:28.561630 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:28.561009 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" podUID="86f1245b-8fd3-4831-abbd-fd9a428ce28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:57:29.343093 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:29.342999 4848 generic.go:356] "Generic (PLEG): container finished" podID="86f1245b-8fd3-4831-abbd-fd9a428ce28b" containerID="97c794d5d63b9970b3f77e92753010da1652eac92eee91ae2be5191082e80651" exitCode=0 Oct 30 20:57:29.343093 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:29.343072 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" event={"ID":"86f1245b-8fd3-4831-abbd-fd9a428ce28b","Type":"ContainerDied","Data":"97c794d5d63b9970b3f77e92753010da1652eac92eee91ae2be5191082e80651"} Oct 30 20:57:29.343528 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:29.343514 4848 scope.go:122] "RemoveContainer" containerID="97c794d5d63b9970b3f77e92753010da1652eac92eee91ae2be5191082e80651" Oct 30 20:57:29.345293 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:29.345265 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:29.345367 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:29.345332 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:29.345404 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:29.345364 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:30.362170 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:30.360654 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8"] Oct 30 20:57:30.362170 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:30.360915 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" podUID="548c8714-1683-4286-865d-96d9d76cd5ed" containerName="istio-proxy" containerID="cri-o://580ecd10875162c4a8a6ba107cc05d2f486bdbf61dd9bc1af60b593af0abf961" gracePeriod=30 Oct 30 20:57:30.376108 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:30.375104 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" event={"ID":"86f1245b-8fd3-4831-abbd-fd9a428ce28b","Type":"ContainerStarted","Data":"4847967d1fb95218b52328c603530895f5f57299a5b0d1f07f420ae9c91dd46a"} Oct 30 20:57:31.045030 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:31.044998 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:57:31.045230 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:31.045123 4848 patch_prober.go:29] interesting pod/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:57:31.045230 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:31.045168 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" podUID="86f1245b-8fd3-4831-abbd-fd9a428ce28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 20:57:31.165110 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:31.161104 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df"] Oct 30 20:57:31.165110 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:31.161385 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df" podUID="e1a9974d-a731-4284-addd-58de76974f30" containerName="payload-pre-processing" containerID="cri-o://19073d11ce0d283f236d328299f7a88e8f1b9d4738babacd6faeffb384aac9b2" gracePeriod=30 Oct 30 20:57:31.448368 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:31.444847 4848 generic.go:356] "Generic (PLEG): container finished" podID="e1a9974d-a731-4284-addd-58de76974f30" containerID="19073d11ce0d283f236d328299f7a88e8f1b9d4738babacd6faeffb384aac9b2" exitCode=0 Oct 30 20:57:31.448368 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:31.446185 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df" event={"ID":"e1a9974d-a731-4284-addd-58de76974f30","Type":"ContainerDied","Data":"19073d11ce0d283f236d328299f7a88e8f1b9d4738babacd6faeffb384aac9b2"} Oct 30 20:57:32.053541 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:32.052970 4848 patch_prober.go:29] interesting pod/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:57:32.053541 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:32.053034 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" podUID="86f1245b-8fd3-4831-abbd-fd9a428ce28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 20:57:32.200858 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:32.196843 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df" Oct 30 20:57:32.328791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:32.328113 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1a9974d-a731-4284-addd-58de76974f30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a9974d-a731-4284-addd-58de76974f30-plugins-config-volume\") pod \"e1a9974d-a731-4284-addd-58de76974f30\" (UID: \"e1a9974d-a731-4284-addd-58de76974f30\") " Oct 30 20:57:32.328791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:32.328173 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1a9974d-a731-4284-addd-58de76974f30-kube-api-access-5dxnj\" (UniqueName: \"kubernetes.io/projected/e1a9974d-a731-4284-addd-58de76974f30-kube-api-access-5dxnj\") pod \"e1a9974d-a731-4284-addd-58de76974f30\" (UID: \"e1a9974d-a731-4284-addd-58de76974f30\") " Oct 30 20:57:32.329273 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:32.329247 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a9974d-a731-4284-addd-58de76974f30-plugins-config-volume" pod "e1a9974d-a731-4284-addd-58de76974f30" (UID: "e1a9974d-a731-4284-addd-58de76974f30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:32.339778 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:32.332089 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a9974d-a731-4284-addd-58de76974f30-kube-api-access-5dxnj" pod "e1a9974d-a731-4284-addd-58de76974f30" (UID: "e1a9974d-a731-4284-addd-58de76974f30"). InnerVolumeSpecName "kube-api-access-5dxnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:32.431280 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:32.431231 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a9974d-a731-4284-addd-58de76974f30-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:32.431280 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:32.431274 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dxnj\" (UniqueName: \"kubernetes.io/projected/e1a9974d-a731-4284-addd-58de76974f30-kube-api-access-5dxnj\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:32.454281 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:32.454199 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df" Oct 30 20:57:32.454281 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:32.454229 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df" event={"ID":"e1a9974d-a731-4284-addd-58de76974f30","Type":"ContainerDied","Data":"9a16a1012cf687238f5145131ff690f9ce889231a0e521ea6c48f477ac0ed842"} Oct 30 20:57:32.454838 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:32.454321 4848 scope.go:122] "RemoveContainer" containerID="19073d11ce0d283f236d328299f7a88e8f1b9d4738babacd6faeffb384aac9b2" Oct 30 20:57:32.479145 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:32.479112 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df"] Oct 30 20:57:32.486490 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:32.486463 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c14e2-6b97c6f9f9-f94df"] Oct 30 20:57:32.788962 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:32.788877 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz"] Oct 30 20:57:32.791293 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:32.790124 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" podUID="bd50e1a1-a095-4b6a-a679-821ef4d48d41" containerName="maas-api" containerID="cri-o://e8afbc4a6e4ac71feaea31aaca439e1537865b70eb1733171e16051ffe4e692e" gracePeriod=30 Oct 30 20:57:32.939938 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:32.939904 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1a9974d-a731-4284-addd-58de76974f30" path="/var/lib/kubelet/pods/e1a9974d-a731-4284-addd-58de76974f30/volumes" Oct 30 20:57:33.048636 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:33.040681 4848 patch_prober.go:29] interesting pod/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:57:33.289795 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:33.040748 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" podUID="86f1245b-8fd3-4831-abbd-fd9a428ce28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 20:57:33.495300 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:33.495265 4848 generic.go:356] "Generic (PLEG): container finished" podID="bd50e1a1-a095-4b6a-a679-821ef4d48d41" containerID="e8afbc4a6e4ac71feaea31aaca439e1537865b70eb1733171e16051ffe4e692e" exitCode=0 Oct 30 20:57:33.495746 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:33.495344 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" event={"ID":"bd50e1a1-a095-4b6a-a679-821ef4d48d41","Type":"ContainerDied","Data":"e8afbc4a6e4ac71feaea31aaca439e1537865b70eb1733171e16051ffe4e692e"} Oct 30 20:57:33.543034 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:33.543008 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" Oct 30 20:57:33.653102 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:33.653074 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd50e1a1-a095-4b6a-a679-821ef4d48d41-kube-api-access-hc595\" (UniqueName: \"kubernetes.io/projected/bd50e1a1-a095-4b6a-a679-821ef4d48d41-kube-api-access-hc595\") pod \"bd50e1a1-a095-4b6a-a679-821ef4d48d41\" (UID: \"bd50e1a1-a095-4b6a-a679-821ef4d48d41\") " Oct 30 20:57:33.653311 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:33.653124 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd50e1a1-a095-4b6a-a679-821ef4d48d41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd50e1a1-a095-4b6a-a679-821ef4d48d41-openshift-service-ca\") pod \"bd50e1a1-a095-4b6a-a679-821ef4d48d41\" (UID: \"bd50e1a1-a095-4b6a-a679-821ef4d48d41\") " Oct 30 20:57:33.653311 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:33.653289 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd50e1a1-a095-4b6a-a679-821ef4d48d41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd50e1a1-a095-4b6a-a679-821ef4d48d41-maas-api-tls\") pod \"bd50e1a1-a095-4b6a-a679-821ef4d48d41\" (UID: \"bd50e1a1-a095-4b6a-a679-821ef4d48d41\") " Oct 30 20:57:33.653454 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:33.653383 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd50e1a1-a095-4b6a-a679-821ef4d48d41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd50e1a1-a095-4b6a-a679-821ef4d48d41-metrics-certs\") pod \"bd50e1a1-a095-4b6a-a679-821ef4d48d41\" (UID: \"bd50e1a1-a095-4b6a-a679-821ef4d48d41\") " Oct 30 20:57:33.653561 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:33.653537 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd50e1a1-a095-4b6a-a679-821ef4d48d41-openshift-service-ca" pod "bd50e1a1-a095-4b6a-a679-821ef4d48d41" (UID: "bd50e1a1-a095-4b6a-a679-821ef4d48d41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:33.654441 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:33.654418 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd50e1a1-a095-4b6a-a679-821ef4d48d41-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:33.655669 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:33.655646 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd50e1a1-a095-4b6a-a679-821ef4d48d41-kube-api-access-hc595" pod "bd50e1a1-a095-4b6a-a679-821ef4d48d41" (UID: "bd50e1a1-a095-4b6a-a679-821ef4d48d41"). InnerVolumeSpecName "kube-api-access-hc595". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:33.655811 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:33.655791 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd50e1a1-a095-4b6a-a679-821ef4d48d41-metrics-certs" pod "bd50e1a1-a095-4b6a-a679-821ef4d48d41" (UID: "bd50e1a1-a095-4b6a-a679-821ef4d48d41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:33.655871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:33.655752 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd50e1a1-a095-4b6a-a679-821ef4d48d41-maas-api-tls" pod "bd50e1a1-a095-4b6a-a679-821ef4d48d41" (UID: "bd50e1a1-a095-4b6a-a679-821ef4d48d41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:57:33.760273 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:33.756320 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd50e1a1-a095-4b6a-a679-821ef4d48d41-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:33.760273 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:33.756366 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd50e1a1-a095-4b6a-a679-821ef4d48d41-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:33.760273 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:33.756383 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc595\" (UniqueName: \"kubernetes.io/projected/bd50e1a1-a095-4b6a-a679-821ef4d48d41-kube-api-access-hc595\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:34.042272 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:34.042184 4848 patch_prober.go:29] interesting pod/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:57:34.042272 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:34.042255 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" podUID="86f1245b-8fd3-4831-abbd-fd9a428ce28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 20:57:34.521244 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:34.518640 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" event={"ID":"bd50e1a1-a095-4b6a-a679-821ef4d48d41","Type":"ContainerDied","Data":"647abdf9b270bd98ed797afa31c4a0a054bff046452e379a455ed685265cf957"} Oct 30 20:57:34.521244 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:34.518715 4848 scope.go:122] "RemoveContainer" containerID="e8afbc4a6e4ac71feaea31aaca439e1537865b70eb1733171e16051ffe4e692e" Oct 30 20:57:34.521244 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:34.518869 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz" Oct 30 20:57:34.557550 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:34.557526 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz"] Oct 30 20:57:34.561899 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:34.561875 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-319188-7b6bcd5789-prptz"] Oct 30 20:57:34.945964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:34.945938 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:57:34.946176 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:34.946158 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" podUID="548c8714-1683-4286-865d-96d9d76cd5ed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:57:34.950079 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:34.949992 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd50e1a1-a095-4b6a-a679-821ef4d48d41" path="/var/lib/kubelet/pods/bd50e1a1-a095-4b6a-a679-821ef4d48d41/volumes" Oct 30 20:57:35.040912 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:35.040875 4848 patch_prober.go:29] interesting pod/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:57:35.041067 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:35.040934 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" podUID="86f1245b-8fd3-4831-abbd-fd9a428ce28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 20:57:35.560239 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:35.560191 4848 generic.go:356] "Generic (PLEG): container finished" podID="548c8714-1683-4286-865d-96d9d76cd5ed" containerID="580ecd10875162c4a8a6ba107cc05d2f486bdbf61dd9bc1af60b593af0abf961" exitCode=0 Oct 30 20:57:35.560712 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:35.560247 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" event={"ID":"548c8714-1683-4286-865d-96d9d76cd5ed","Type":"ContainerDied","Data":"580ecd10875162c4a8a6ba107cc05d2f486bdbf61dd9bc1af60b593af0abf961"} Oct 30 20:57:35.560712 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:35.560334 4848 scope.go:122] "RemoveContainer" containerID="d965eb356a54058eb8f35428787584c97c196c994ae44c1d0ecb2679ba1d286d" Oct 30 20:57:36.042901 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.042868 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:57:36.054101 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.054016 4848 patch_prober.go:29] interesting pod/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 20:57:36.054535 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.054505 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" podUID="86f1245b-8fd3-4831-abbd-fd9a428ce28b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 20:57:36.255938 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.255896 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-workload-certs\") pod \"548c8714-1683-4286-865d-96d9d76cd5ed\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " Oct 30 20:57:36.285871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.280259 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/548c8714-1683-4286-865d-96d9d76cd5ed-kube-api-access-5kjhg\" (UniqueName: \"kubernetes.io/projected/548c8714-1683-4286-865d-96d9d76cd5ed-kube-api-access-5kjhg\") pod \"548c8714-1683-4286-865d-96d9d76cd5ed\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " Oct 30 20:57:36.285871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.280332 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-istio-data\") pod \"548c8714-1683-4286-865d-96d9d76cd5ed\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " Oct 30 20:57:36.285871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.280462 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-istio-envoy\") pod \"548c8714-1683-4286-865d-96d9d76cd5ed\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " Oct 30 20:57:36.285871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.280547 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/548c8714-1683-4286-865d-96d9d76cd5ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/548c8714-1683-4286-865d-96d9d76cd5ed-istiod-ca-cert\") pod \"548c8714-1683-4286-865d-96d9d76cd5ed\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " Oct 30 20:57:36.285871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.280648 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/548c8714-1683-4286-865d-96d9d76cd5ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/548c8714-1683-4286-865d-96d9d76cd5ed-istio-podinfo\") pod \"548c8714-1683-4286-865d-96d9d76cd5ed\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " Oct 30 20:57:36.285871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.280732 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/548c8714-1683-4286-865d-96d9d76cd5ed-istio-token\" (UniqueName: \"kubernetes.io/projected/548c8714-1683-4286-865d-96d9d76cd5ed-istio-token\") pod \"548c8714-1683-4286-865d-96d9d76cd5ed\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " Oct 30 20:57:36.285871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.277170 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-workload-certs" pod "548c8714-1683-4286-865d-96d9d76cd5ed" (UID: "548c8714-1683-4286-865d-96d9d76cd5ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:36.285871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.280912 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-istio-data" pod "548c8714-1683-4286-865d-96d9d76cd5ed" (UID: "548c8714-1683-4286-865d-96d9d76cd5ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:36.285871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.280940 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-workload-socket\") pod \"548c8714-1683-4286-865d-96d9d76cd5ed\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " Oct 30 20:57:36.285871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.281048 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-credential-socket\") pod \"548c8714-1683-4286-865d-96d9d76cd5ed\" (UID: \"548c8714-1683-4286-865d-96d9d76cd5ed\") " Oct 30 20:57:36.285871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.281416 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-workload-socket" pod "548c8714-1683-4286-865d-96d9d76cd5ed" (UID: "548c8714-1683-4286-865d-96d9d76cd5ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:36.285871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.281643 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-credential-socket" pod "548c8714-1683-4286-865d-96d9d76cd5ed" (UID: "548c8714-1683-4286-865d-96d9d76cd5ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:36.285871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.282205 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:36.285871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.282226 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:36.285871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.282254 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:36.285871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.282270 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:36.302859 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.302822 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548c8714-1683-4286-865d-96d9d76cd5ed-kube-api-access-5kjhg" pod "548c8714-1683-4286-865d-96d9d76cd5ed" (UID: "548c8714-1683-4286-865d-96d9d76cd5ed"). InnerVolumeSpecName "kube-api-access-5kjhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:36.308539 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.307208 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548c8714-1683-4286-865d-96d9d76cd5ed-istio-token" pod "548c8714-1683-4286-865d-96d9d76cd5ed" (UID: "548c8714-1683-4286-865d-96d9d76cd5ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:36.309669 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.309645 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-istio-envoy" pod "548c8714-1683-4286-865d-96d9d76cd5ed" (UID: "548c8714-1683-4286-865d-96d9d76cd5ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:36.311035 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.310935 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/548c8714-1683-4286-865d-96d9d76cd5ed-istio-podinfo" pod "548c8714-1683-4286-865d-96d9d76cd5ed" (UID: "548c8714-1683-4286-865d-96d9d76cd5ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:57:36.311035 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.310989 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/548c8714-1683-4286-865d-96d9d76cd5ed-istiod-ca-cert" pod "548c8714-1683-4286-865d-96d9d76cd5ed" (UID: "548c8714-1683-4286-865d-96d9d76cd5ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:36.384752 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.384710 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kjhg\" (UniqueName: \"kubernetes.io/projected/548c8714-1683-4286-865d-96d9d76cd5ed-kube-api-access-5kjhg\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:36.384752 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.384753 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/548c8714-1683-4286-865d-96d9d76cd5ed-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:36.384752 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.384784 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/548c8714-1683-4286-865d-96d9d76cd5ed-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:36.385032 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.384799 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/548c8714-1683-4286-865d-96d9d76cd5ed-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:36.385032 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.384813 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/548c8714-1683-4286-865d-96d9d76cd5ed-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:36.568873 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.568757 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" Oct 30 20:57:36.568873 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.568814 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8" event={"ID":"548c8714-1683-4286-865d-96d9d76cd5ed","Type":"ContainerDied","Data":"7c6dd11495c83d8f85de17ee42730e64cee7e67e0d7cb59e65f3d8ef40261330"} Oct 30 20:57:36.569437 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.568887 4848 scope.go:122] "RemoveContainer" containerID="580ecd10875162c4a8a6ba107cc05d2f486bdbf61dd9bc1af60b593af0abf961" Oct 30 20:57:36.594366 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.594336 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8"] Oct 30 20:57:36.599185 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.599158 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-eebcb4-openshift-default-74d678989-7skb8"] Oct 30 20:57:36.933695 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:36.933661 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="548c8714-1683-4286-865d-96d9d76cd5ed" path="/var/lib/kubelet/pods/548c8714-1683-4286-865d-96d9d76cd5ed/volumes" Oct 30 20:57:37.045706 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:37.045676 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:57:37.046126 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:37.046103 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:57:37.046705 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:37.046686 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:57:39.645096 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:39.645057 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz"] Oct 30 20:57:39.645605 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:39.645332 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" podUID="b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" containerName="istio-proxy" containerID="cri-o://da481c7d2d6eeea25a8fc53a5bc200621989ada819d822d19f8ec19575eed0fb" gracePeriod=30 Oct 30 20:57:41.310245 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:41.310208 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5"] Oct 30 20:57:41.310741 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:41.310482 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" podUID="86f1245b-8fd3-4831-abbd-fd9a428ce28b" containerName="istio-proxy" containerID="cri-o://4847967d1fb95218b52328c603530895f5f57299a5b0d1f07f420ae9c91dd46a" gracePeriod=30 Oct 30 20:57:43.503611 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:43.503575 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx"] Oct 30 20:57:43.504115 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:43.503866 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" podUID="2d0f604c-f9f8-40fb-ab5e-8517520166a3" containerName="istio-proxy" containerID="cri-o://0535ab83e33041130a6aa473ff5cdaf52048834874a057561e502aff896917ba" gracePeriod=30 Oct 30 20:57:44.940101 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:44.940070 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:57:45.002244 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.002162 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-envoy\") pod \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " Oct 30 20:57:45.002442 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.002292 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-workload-socket\") pod \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " Oct 30 20:57:45.002442 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.002342 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-workload-certs\") pod \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " Oct 30 20:57:45.002442 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.002395 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istiod-ca-cert\") pod \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " Oct 30 20:57:45.002752 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.002442 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-kube-api-access-xmhcx\" (UniqueName: \"kubernetes.io/projected/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-kube-api-access-xmhcx\") pod \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " Oct 30 20:57:45.002752 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.002509 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-data\") pod \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " Oct 30 20:57:45.002752 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.002565 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-podinfo\") pod \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " Oct 30 20:57:45.002752 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.002605 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-token\" (UniqueName: \"kubernetes.io/projected/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-token\") pod \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " Oct 30 20:57:45.002752 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.002632 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-credential-socket\") pod \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\" (UID: \"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f\") " Oct 30 20:57:45.003039 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.002761 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-workload-certs" pod "b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" (UID: "b2e499bb-0b7e-4bb6-9966-2ac87b0c449f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:45.003039 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.002947 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-data" pod "b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" (UID: "b2e499bb-0b7e-4bb6-9966-2ac87b0c449f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:45.003039 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.002977 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-workload-socket" pod "b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" (UID: "b2e499bb-0b7e-4bb6-9966-2ac87b0c449f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:45.003227 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.003202 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istiod-ca-cert" pod "b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" (UID: "b2e499bb-0b7e-4bb6-9966-2ac87b0c449f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:45.003286 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.003228 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-credential-socket" pod "b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" (UID: "b2e499bb-0b7e-4bb6-9966-2ac87b0c449f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:45.003730 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.003707 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:45.003862 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.003735 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:45.003862 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.003753 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:45.003862 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.003783 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:45.003862 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.003796 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:45.005231 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.005207 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-kube-api-access-xmhcx" pod "b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" (UID: "b2e499bb-0b7e-4bb6-9966-2ac87b0c449f"). InnerVolumeSpecName "kube-api-access-xmhcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:45.005858 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.005828 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-envoy" pod "b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" (UID: "b2e499bb-0b7e-4bb6-9966-2ac87b0c449f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:45.005858 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.005852 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-podinfo" pod "b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" (UID: "b2e499bb-0b7e-4bb6-9966-2ac87b0c449f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:57:45.006013 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.005998 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-token" pod "b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" (UID: "b2e499bb-0b7e-4bb6-9966-2ac87b0c449f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:45.127308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.127269 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:45.127624 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.127589 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:45.127814 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.127795 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:45.128020 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.128002 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmhcx\" (UniqueName: \"kubernetes.io/projected/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f-kube-api-access-xmhcx\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:45.465790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.465746 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m"] Oct 30 20:57:45.466738 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.466696 4848 cpu_manager.go:414] "containerMap: removing container" podUID="548c8714-1683-4286-865d-96d9d76cd5ed" containerName="istio-proxy" Oct 30 20:57:45.466910 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.466894 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="548c8714-1683-4286-865d-96d9d76cd5ed" containerName="istio-proxy" Oct 30 20:57:45.467014 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467000 4848 cpu_manager.go:414] "containerMap: removing container" podUID="07654e83-e9e1-4297-be2b-41591a932b39" containerName="maas-api" Oct 30 20:57:45.467092 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467014 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07654e83-e9e1-4297-be2b-41591a932b39" containerName="maas-api" Oct 30 20:57:45.467092 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467027 4848 cpu_manager.go:414] "containerMap: removing container" podUID="b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" containerName="istio-proxy" Oct 30 20:57:45.467092 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467035 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" containerName="istio-proxy" Oct 30 20:57:45.467092 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467048 4848 cpu_manager.go:414] "containerMap: removing container" podUID="bf9525d3-a53f-4e30-8fae-12a407eace4e" containerName="istio-proxy" Oct 30 20:57:45.467092 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467056 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf9525d3-a53f-4e30-8fae-12a407eace4e" containerName="istio-proxy" Oct 30 20:57:45.467341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467106 4848 cpu_manager.go:414] "containerMap: removing container" podUID="bd50e1a1-a095-4b6a-a679-821ef4d48d41" containerName="maas-api" Oct 30 20:57:45.467341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467114 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd50e1a1-a095-4b6a-a679-821ef4d48d41" containerName="maas-api" Oct 30 20:57:45.467341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467126 4848 cpu_manager.go:414] "containerMap: removing container" podUID="849dd3a0-afee-4409-be49-d0a264b2b31b" containerName="maas-api" Oct 30 20:57:45.467341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467132 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="849dd3a0-afee-4409-be49-d0a264b2b31b" containerName="maas-api" Oct 30 20:57:45.467341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467151 4848 cpu_manager.go:414] "containerMap: removing container" podUID="bf9525d3-a53f-4e30-8fae-12a407eace4e" containerName="istio-proxy" Oct 30 20:57:45.467341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467158 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf9525d3-a53f-4e30-8fae-12a407eace4e" containerName="istio-proxy" Oct 30 20:57:45.467341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467174 4848 cpu_manager.go:414] "containerMap: removing container" podUID="b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" containerName="istio-proxy" Oct 30 20:57:45.467341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467182 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" containerName="istio-proxy" Oct 30 20:57:45.467341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467193 4848 cpu_manager.go:414] "containerMap: removing container" podUID="9f9f3012-8124-49be-92c4-3f04feb2a82c" containerName="payload-pre-processing" Oct 30 20:57:45.467341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467200 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f9f3012-8124-49be-92c4-3f04feb2a82c" containerName="payload-pre-processing" Oct 30 20:57:45.467341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467217 4848 cpu_manager.go:414] "containerMap: removing container" podUID="548c8714-1683-4286-865d-96d9d76cd5ed" containerName="istio-proxy" Oct 30 20:57:45.467341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467225 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="548c8714-1683-4286-865d-96d9d76cd5ed" containerName="istio-proxy" Oct 30 20:57:45.467341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467238 4848 cpu_manager.go:414] "containerMap: removing container" podUID="e1a9974d-a731-4284-addd-58de76974f30" containerName="payload-pre-processing" Oct 30 20:57:45.467341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467246 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1a9974d-a731-4284-addd-58de76974f30" containerName="payload-pre-processing" Oct 30 20:57:45.468127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467407 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd50e1a1-a095-4b6a-a679-821ef4d48d41" containerName="maas-api" Oct 30 20:57:45.468127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467423 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="849dd3a0-afee-4409-be49-d0a264b2b31b" containerName="maas-api" Oct 30 20:57:45.468127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467435 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf9525d3-a53f-4e30-8fae-12a407eace4e" containerName="istio-proxy" Oct 30 20:57:45.468127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467447 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" containerName="istio-proxy" Oct 30 20:57:45.468127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467456 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f9f3012-8124-49be-92c4-3f04feb2a82c" containerName="payload-pre-processing" Oct 30 20:57:45.468127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467467 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="548c8714-1683-4286-865d-96d9d76cd5ed" containerName="istio-proxy" Oct 30 20:57:45.468127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467476 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1a9974d-a731-4284-addd-58de76974f30" containerName="payload-pre-processing" Oct 30 20:57:45.468127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467488 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="548c8714-1683-4286-865d-96d9d76cd5ed" containerName="istio-proxy" Oct 30 20:57:45.468127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467499 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="07654e83-e9e1-4297-be2b-41591a932b39" containerName="maas-api" Oct 30 20:57:45.468127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467509 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" containerName="istio-proxy" Oct 30 20:57:45.468127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.467520 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf9525d3-a53f-4e30-8fae-12a407eace4e" containerName="istio-proxy" Oct 30 20:57:45.471710 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.471687 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.483163 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.480670 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-2b7ca7-openshift-default-dockercfg-5jd6f\"" Oct 30 20:57:45.483163 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.483085 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m"] Oct 30 20:57:45.534571 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.534533 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g7fb\" (UniqueName: \"kubernetes.io/projected/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-kube-api-access-8g7fb\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.534827 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.534740 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-credential-socket\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.534827 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.534786 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-workload-certs\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.535084 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.535035 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istiod-ca-cert\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.535176 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.535083 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-data\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.535176 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.535118 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-podinfo\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.535176 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.535141 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-token\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.535176 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.535167 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-envoy\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.535434 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.535191 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-workload-socket\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.649097 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.648983 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-credential-socket\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.649097 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.649056 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-workload-certs\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.649643 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.649607 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istiod-ca-cert\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.679145 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.679086 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-data\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.679526 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.679428 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-podinfo\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.679526 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.679476 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-token\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.679526 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.679502 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-envoy\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.679834 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.679794 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-workload-socket\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.680064 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.680017 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g7fb\" (UniqueName: \"kubernetes.io/projected/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-kube-api-access-8g7fb\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.680867 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.658333 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-workload-certs\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.686440 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.686408 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-data\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.686611 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.657710 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istiod-ca-cert\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.686905 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.658085 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-credential-socket\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.687059 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.686967 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-workload-socket\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.706972 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.691107 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-envoy\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.724877 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.718313 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-podinfo\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.729439 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.729408 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-token\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.730321 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.730296 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g7fb\" (UniqueName: \"kubernetes.io/projected/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-kube-api-access-8g7fb\") pod \"e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.813788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.792910 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:45.862818 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.860650 4848 generic.go:356] "Generic (PLEG): container finished" podID="b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" containerID="da481c7d2d6eeea25a8fc53a5bc200621989ada819d822d19f8ec19575eed0fb" exitCode=0 Oct 30 20:57:45.862818 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.860752 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" event={"ID":"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f","Type":"ContainerDied","Data":"da481c7d2d6eeea25a8fc53a5bc200621989ada819d822d19f8ec19575eed0fb"} Oct 30 20:57:45.883797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.871065 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" event={"ID":"b2e499bb-0b7e-4bb6-9966-2ac87b0c449f","Type":"ContainerDied","Data":"3aa3b08ee995b1c220a22cbb0b223db60582aa5632e9aec64bce49fe55f6cc3b"} Oct 30 20:57:45.883797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.871121 4848 scope.go:122] "RemoveContainer" containerID="da481c7d2d6eeea25a8fc53a5bc200621989ada819d822d19f8ec19575eed0fb" Oct 30 20:57:45.883797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.869879 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz" Oct 30 20:57:45.955461 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.955434 4848 scope.go:122] "RemoveContainer" containerID="9743ef69836b78a654e81a7b5066df32e4c72851bd8fe21d78d647757db10a62" Oct 30 20:57:45.995688 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.993849 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz"] Oct 30 20:57:45.996702 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:45.996605 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-7c14e2-openshift-default-68fdcfcc6c-kpkqz"] Oct 30 20:57:46.004252 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.004229 4848 scope.go:122] "RemoveContainer" containerID="da481c7d2d6eeea25a8fc53a5bc200621989ada819d822d19f8ec19575eed0fb" Oct 30 20:57:46.005291 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:57:46.005266 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da481c7d2d6eeea25a8fc53a5bc200621989ada819d822d19f8ec19575eed0fb\": container with ID starting with da481c7d2d6eeea25a8fc53a5bc200621989ada819d822d19f8ec19575eed0fb not found: ID does not exist" containerID="da481c7d2d6eeea25a8fc53a5bc200621989ada819d822d19f8ec19575eed0fb" Oct 30 20:57:46.005386 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.005302 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da481c7d2d6eeea25a8fc53a5bc200621989ada819d822d19f8ec19575eed0fb"} err="failed to get container status \"da481c7d2d6eeea25a8fc53a5bc200621989ada819d822d19f8ec19575eed0fb\": rpc error: code = NotFound desc = could not find container \"da481c7d2d6eeea25a8fc53a5bc200621989ada819d822d19f8ec19575eed0fb\": container with ID starting with da481c7d2d6eeea25a8fc53a5bc200621989ada819d822d19f8ec19575eed0fb not found: ID does not exist" Oct 30 20:57:46.005386 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.005327 4848 scope.go:122] "RemoveContainer" containerID="9743ef69836b78a654e81a7b5066df32e4c72851bd8fe21d78d647757db10a62" Oct 30 20:57:46.005890 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:57:46.005864 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9743ef69836b78a654e81a7b5066df32e4c72851bd8fe21d78d647757db10a62\": container with ID starting with 9743ef69836b78a654e81a7b5066df32e4c72851bd8fe21d78d647757db10a62 not found: ID does not exist" containerID="9743ef69836b78a654e81a7b5066df32e4c72851bd8fe21d78d647757db10a62" Oct 30 20:57:46.005958 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.005930 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9743ef69836b78a654e81a7b5066df32e4c72851bd8fe21d78d647757db10a62"} err="failed to get container status \"9743ef69836b78a654e81a7b5066df32e4c72851bd8fe21d78d647757db10a62\": rpc error: code = NotFound desc = could not find container \"9743ef69836b78a654e81a7b5066df32e4c72851bd8fe21d78d647757db10a62\": container with ID starting with 9743ef69836b78a654e81a7b5066df32e4c72851bd8fe21d78d647757db10a62 not found: ID does not exist" Oct 30 20:57:46.121135 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.121099 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx"] Oct 30 20:57:46.121374 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.121330 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx" podUID="9ca3d842-f891-4981-a687-6971c181b6b7" containerName="payload-processing" containerID="cri-o://383fe338c7397ef533b04e645bc075e37c5b3475823f1dc3d6fffe9023eedb98" gracePeriod=30 Oct 30 20:57:46.127175 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.126452 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c"] Oct 30 20:57:46.127175 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.126670 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c" podUID="0d25fd62-4190-4453-bf50-38d631d53cf1" containerName="payload-pre-processing" containerID="cri-o://dd4360eeaf2d216c9e993a3287dfaeb85de6faffaa9678ba77035170a0c34288" gracePeriod=30 Oct 30 20:57:46.255801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.255752 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m"] Oct 30 20:57:46.257079 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:57:46.257049 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e85ca9_98d4_4a78_a5c6_a50d5d2cad80.slice/crio-1c72d2dcbc99e93b562755913429435d779da6a8629fbff6f165337dee68302f WatchSource:0}: Error finding container 1c72d2dcbc99e93b562755913429435d779da6a8629fbff6f165337dee68302f: Status 404 returned error can't find the container with id 1c72d2dcbc99e93b562755913429435d779da6a8629fbff6f165337dee68302f Oct 30 20:57:46.263944 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.263908 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:46.264077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.264002 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:46.264077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.264034 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:46.830638 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.812527 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx" Oct 30 20:57:46.999808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.984887 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e499bb-0b7e-4bb6-9966-2ac87b0c449f" path="/var/lib/kubelet/pods/b2e499bb-0b7e-4bb6-9966-2ac87b0c449f/volumes" Oct 30 20:57:46.999808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.989524 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" event={"ID":"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80","Type":"ContainerStarted","Data":"9fcb9d3e9c7a735ea417b94694a721a519405bdeb6be2be0ad24e54d188492f5"} Oct 30 20:57:46.999808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.989577 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" event={"ID":"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80","Type":"ContainerStarted","Data":"1c72d2dcbc99e93b562755913429435d779da6a8629fbff6f165337dee68302f"} Oct 30 20:57:46.999808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.991694 4848 generic.go:356] "Generic (PLEG): container finished" podID="0d25fd62-4190-4453-bf50-38d631d53cf1" containerID="dd4360eeaf2d216c9e993a3287dfaeb85de6faffaa9678ba77035170a0c34288" exitCode=0 Oct 30 20:57:46.999808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.991760 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c" event={"ID":"0d25fd62-4190-4453-bf50-38d631d53cf1","Type":"ContainerDied","Data":"dd4360eeaf2d216c9e993a3287dfaeb85de6faffaa9678ba77035170a0c34288"} Oct 30 20:57:46.999808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.994203 4848 generic.go:356] "Generic (PLEG): container finished" podID="9ca3d842-f891-4981-a687-6971c181b6b7" containerID="383fe338c7397ef533b04e645bc075e37c5b3475823f1dc3d6fffe9023eedb98" exitCode=0 Oct 30 20:57:46.999808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.994345 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx" event={"ID":"9ca3d842-f891-4981-a687-6971c181b6b7","Type":"ContainerDied","Data":"383fe338c7397ef533b04e645bc075e37c5b3475823f1dc3d6fffe9023eedb98"} Oct 30 20:57:46.999808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.994382 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx" event={"ID":"9ca3d842-f891-4981-a687-6971c181b6b7","Type":"ContainerDied","Data":"b6a1a662f3bee82f46502f2bf21c4206e02b2703ea89ab520f39bba5599892cd"} Oct 30 20:57:46.999808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.994411 4848 scope.go:122] "RemoveContainer" containerID="383fe338c7397ef533b04e645bc075e37c5b3475823f1dc3d6fffe9023eedb98" Oct 30 20:57:46.999808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.994501 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx" Oct 30 20:57:46.999808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.997795 4848 generic.go:356] "Generic (PLEG): container finished" podID="86f1245b-8fd3-4831-abbd-fd9a428ce28b" containerID="4847967d1fb95218b52328c603530895f5f57299a5b0d1f07f420ae9c91dd46a" exitCode=0 Oct 30 20:57:46.999808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:46.997837 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" event={"ID":"86f1245b-8fd3-4831-abbd-fd9a428ce28b","Type":"ContainerDied","Data":"4847967d1fb95218b52328c603530895f5f57299a5b0d1f07f420ae9c91dd46a"} Oct 30 20:57:47.018599 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.018573 4848 scope.go:122] "RemoveContainer" containerID="383fe338c7397ef533b04e645bc075e37c5b3475823f1dc3d6fffe9023eedb98" Oct 30 20:57:47.019021 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:57:47.018981 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383fe338c7397ef533b04e645bc075e37c5b3475823f1dc3d6fffe9023eedb98\": container with ID starting with 383fe338c7397ef533b04e645bc075e37c5b3475823f1dc3d6fffe9023eedb98 not found: ID does not exist" containerID="383fe338c7397ef533b04e645bc075e37c5b3475823f1dc3d6fffe9023eedb98" Oct 30 20:57:47.019163 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.019033 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383fe338c7397ef533b04e645bc075e37c5b3475823f1dc3d6fffe9023eedb98"} err="failed to get container status \"383fe338c7397ef533b04e645bc075e37c5b3475823f1dc3d6fffe9023eedb98\": rpc error: code = NotFound desc = could not find container \"383fe338c7397ef533b04e645bc075e37c5b3475823f1dc3d6fffe9023eedb98\": container with ID starting with 383fe338c7397ef533b04e645bc075e37c5b3475823f1dc3d6fffe9023eedb98 not found: ID does not exist" Oct 30 20:57:47.019163 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.019055 4848 scope.go:122] "RemoveContainer" containerID="97c794d5d63b9970b3f77e92753010da1652eac92eee91ae2be5191082e80651" Oct 30 20:57:47.019346 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.019300 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" podStartSLOduration=2.019283508 podStartE2EDuration="2.019283508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:47.013386731 +0000 UTC m=+1172.193283239" watchObservedRunningTime="2026-10-30 20:57:47.019283508 +0000 UTC m=+1172.199180018" Oct 30 20:57:47.067603 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.067500 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca3d842-f891-4981-a687-6971c181b6b7-kube-api-access-rvn2z\" (UniqueName: \"kubernetes.io/projected/9ca3d842-f891-4981-a687-6971c181b6b7-kube-api-access-rvn2z\") pod \"9ca3d842-f891-4981-a687-6971c181b6b7\" (UID: \"9ca3d842-f891-4981-a687-6971c181b6b7\") " Oct 30 20:57:47.068471 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.068376 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ca3d842-f891-4981-a687-6971c181b6b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca3d842-f891-4981-a687-6971c181b6b7-plugins-config-volume\") pod \"9ca3d842-f891-4981-a687-6971c181b6b7\" (UID: \"9ca3d842-f891-4981-a687-6971c181b6b7\") " Oct 30 20:57:47.069219 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.068725 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca3d842-f891-4981-a687-6971c181b6b7-plugins-config-volume" pod "9ca3d842-f891-4981-a687-6971c181b6b7" (UID: "9ca3d842-f891-4981-a687-6971c181b6b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:47.069704 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.069682 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ca3d842-f891-4981-a687-6971c181b6b7-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.070816 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.070749 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca3d842-f891-4981-a687-6971c181b6b7-kube-api-access-rvn2z" pod "9ca3d842-f891-4981-a687-6971c181b6b7" (UID: "9ca3d842-f891-4981-a687-6971c181b6b7"). InnerVolumeSpecName "kube-api-access-rvn2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:47.078912 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.078883 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c" Oct 30 20:57:47.103259 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.103230 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:57:47.172792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.170621 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-envoy\") pod \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " Oct 30 20:57:47.172792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.170672 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-workload-certs\") pod \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " Oct 30 20:57:47.172792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.170705 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d25fd62-4190-4453-bf50-38d631d53cf1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d25fd62-4190-4453-bf50-38d631d53cf1-plugins-config-volume\") pod \"0d25fd62-4190-4453-bf50-38d631d53cf1\" (UID: \"0d25fd62-4190-4453-bf50-38d631d53cf1\") " Oct 30 20:57:47.172792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.170743 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d25fd62-4190-4453-bf50-38d631d53cf1-kube-api-access-fcs2m\" (UniqueName: \"kubernetes.io/projected/0d25fd62-4190-4453-bf50-38d631d53cf1-kube-api-access-fcs2m\") pod \"0d25fd62-4190-4453-bf50-38d631d53cf1\" (UID: \"0d25fd62-4190-4453-bf50-38d631d53cf1\") " Oct 30 20:57:47.172792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.170817 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-workload-socket\") pod \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " Oct 30 20:57:47.172792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.170861 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-token\" (UniqueName: \"kubernetes.io/projected/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-token\") pod \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " Oct 30 20:57:47.172792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.170918 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86f1245b-8fd3-4831-abbd-fd9a428ce28b-kube-api-access-5qv9p\" (UniqueName: \"kubernetes.io/projected/86f1245b-8fd3-4831-abbd-fd9a428ce28b-kube-api-access-5qv9p\") pod \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " Oct 30 20:57:47.172792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.170970 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-credential-socket\") pod \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " Oct 30 20:57:47.172792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.171096 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-podinfo\") pod \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " Oct 30 20:57:47.172792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.171151 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istiod-ca-cert\") pod \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " Oct 30 20:57:47.172792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.171348 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-data\") pod \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\" (UID: \"86f1245b-8fd3-4831-abbd-fd9a428ce28b\") " Oct 30 20:57:47.172792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.171972 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvn2z\" (UniqueName: \"kubernetes.io/projected/9ca3d842-f891-4981-a687-6971c181b6b7-kube-api-access-rvn2z\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.172792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.172280 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-data" pod "86f1245b-8fd3-4831-abbd-fd9a428ce28b" (UID: "86f1245b-8fd3-4831-abbd-fd9a428ce28b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:47.175047 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.174755 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-workload-certs" pod "86f1245b-8fd3-4831-abbd-fd9a428ce28b" (UID: "86f1245b-8fd3-4831-abbd-fd9a428ce28b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:47.175381 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.175346 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-workload-socket" pod "86f1245b-8fd3-4831-abbd-fd9a428ce28b" (UID: "86f1245b-8fd3-4831-abbd-fd9a428ce28b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:47.175622 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.175601 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-credential-socket" pod "86f1245b-8fd3-4831-abbd-fd9a428ce28b" (UID: "86f1245b-8fd3-4831-abbd-fd9a428ce28b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:47.175994 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.175952 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istiod-ca-cert" pod "86f1245b-8fd3-4831-abbd-fd9a428ce28b" (UID: "86f1245b-8fd3-4831-abbd-fd9a428ce28b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:47.176476 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.176455 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-envoy" pod "86f1245b-8fd3-4831-abbd-fd9a428ce28b" (UID: "86f1245b-8fd3-4831-abbd-fd9a428ce28b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:47.177694 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.177669 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d25fd62-4190-4453-bf50-38d631d53cf1-plugins-config-volume" pod "0d25fd62-4190-4453-bf50-38d631d53cf1" (UID: "0d25fd62-4190-4453-bf50-38d631d53cf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:47.180322 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.180293 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-podinfo" pod "86f1245b-8fd3-4831-abbd-fd9a428ce28b" (UID: "86f1245b-8fd3-4831-abbd-fd9a428ce28b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:57:47.180475 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.180452 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f1245b-8fd3-4831-abbd-fd9a428ce28b-kube-api-access-5qv9p" pod "86f1245b-8fd3-4831-abbd-fd9a428ce28b" (UID: "86f1245b-8fd3-4831-abbd-fd9a428ce28b"). InnerVolumeSpecName "kube-api-access-5qv9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:47.181073 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.181052 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-token" pod "86f1245b-8fd3-4831-abbd-fd9a428ce28b" (UID: "86f1245b-8fd3-4831-abbd-fd9a428ce28b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:47.186128 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.186087 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d25fd62-4190-4453-bf50-38d631d53cf1-kube-api-access-fcs2m" pod "0d25fd62-4190-4453-bf50-38d631d53cf1" (UID: "0d25fd62-4190-4453-bf50-38d631d53cf1"). InnerVolumeSpecName "kube-api-access-fcs2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:47.276023 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.274884 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.276023 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.274923 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.276023 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.274936 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d25fd62-4190-4453-bf50-38d631d53cf1-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.276023 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.274950 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcs2m\" (UniqueName: \"kubernetes.io/projected/0d25fd62-4190-4453-bf50-38d631d53cf1-kube-api-access-fcs2m\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.276023 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.274964 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.276023 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.274975 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.276023 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.274988 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qv9p\" (UniqueName: \"kubernetes.io/projected/86f1245b-8fd3-4831-abbd-fd9a428ce28b-kube-api-access-5qv9p\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.276023 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.275000 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.276023 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.275012 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.276023 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.275023 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.276023 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.275036 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86f1245b-8fd3-4831-abbd-fd9a428ce28b-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:47.330160 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.330127 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx"] Oct 30 20:57:47.336038 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.336004 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a61855-59c8f6c6dd-xlrrx"] Oct 30 20:57:47.803595 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.803566 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:47.808891 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:47.808867 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:48.013987 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:48.013906 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" Oct 30 20:57:48.013987 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:48.013940 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5" event={"ID":"86f1245b-8fd3-4831-abbd-fd9a428ce28b","Type":"ContainerDied","Data":"6789aef48f03b0e100816c5b10460daa7648601cda4bf4f55db97bcc0c90b4d3"} Oct 30 20:57:48.014491 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:48.014028 4848 scope.go:122] "RemoveContainer" containerID="4847967d1fb95218b52328c603530895f5f57299a5b0d1f07f420ae9c91dd46a" Oct 30 20:57:48.015723 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:48.015708 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c" Oct 30 20:57:48.015944 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:48.015731 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c" event={"ID":"0d25fd62-4190-4453-bf50-38d631d53cf1","Type":"ContainerDied","Data":"da2e27c7c917a981f906502365c96a8acaa4bc64ca8e714c9b1db1918b2afaea"} Oct 30 20:57:48.017369 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:48.017347 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:48.018436 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:48.018416 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:57:48.030825 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:48.030808 4848 scope.go:122] "RemoveContainer" containerID="dd4360eeaf2d216c9e993a3287dfaeb85de6faffaa9678ba77035170a0c34288" Oct 30 20:57:48.064082 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:48.064016 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5"] Oct 30 20:57:48.073600 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:48.073555 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-319188-openshift-default-5bb45bd965-n24w5"] Oct 30 20:57:48.086706 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:48.086681 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c"] Oct 30 20:57:48.094944 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:48.094919 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a61855-6c677847-tl98c"] Oct 30 20:57:48.795233 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:48.795166 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:57:48.979297 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:48.979264 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d25fd62-4190-4453-bf50-38d631d53cf1" path="/var/lib/kubelet/pods/0d25fd62-4190-4453-bf50-38d631d53cf1/volumes" Oct 30 20:57:48.980376 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:48.980352 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86f1245b-8fd3-4831-abbd-fd9a428ce28b" path="/var/lib/kubelet/pods/86f1245b-8fd3-4831-abbd-fd9a428ce28b/volumes" Oct 30 20:57:48.981127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:48.981109 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ca3d842-f891-4981-a687-6971c181b6b7" path="/var/lib/kubelet/pods/9ca3d842-f891-4981-a687-6971c181b6b7/volumes" Oct 30 20:57:49.028682 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.028648 4848 generic.go:356] "Generic (PLEG): container finished" podID="2d0f604c-f9f8-40fb-ab5e-8517520166a3" containerID="0535ab83e33041130a6aa473ff5cdaf52048834874a057561e502aff896917ba" exitCode=0 Oct 30 20:57:49.029202 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.028702 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" Oct 30 20:57:49.029202 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.028723 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" event={"ID":"2d0f604c-f9f8-40fb-ab5e-8517520166a3","Type":"ContainerDied","Data":"0535ab83e33041130a6aa473ff5cdaf52048834874a057561e502aff896917ba"} Oct 30 20:57:49.029202 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.028788 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx" event={"ID":"2d0f604c-f9f8-40fb-ab5e-8517520166a3","Type":"ContainerDied","Data":"548ee4cd745ae21069ec48c1db1d35f30b9b0c34cc4220dd2e1f4f212885a2a4"} Oct 30 20:57:49.029202 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.028813 4848 scope.go:122] "RemoveContainer" containerID="0535ab83e33041130a6aa473ff5cdaf52048834874a057561e502aff896917ba" Oct 30 20:57:49.038867 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.038844 4848 scope.go:122] "RemoveContainer" containerID="8c4aa4c4e8cf753eeceea15dea7b0bc657490cd1de2b382fa629f05006d908b1" Oct 30 20:57:49.039366 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.039344 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istiod-ca-cert\") pod \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " Oct 30 20:57:49.039478 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.039397 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-token\" (UniqueName: \"kubernetes.io/projected/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-token\") pod \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " Oct 30 20:57:49.039552 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.039491 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-envoy\") pod \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " Oct 30 20:57:49.039614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.039578 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d0f604c-f9f8-40fb-ab5e-8517520166a3-kube-api-access-vg6cp\" (UniqueName: \"kubernetes.io/projected/2d0f604c-f9f8-40fb-ab5e-8517520166a3-kube-api-access-vg6cp\") pod \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " Oct 30 20:57:49.039671 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.039641 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-podinfo\") pod \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " Oct 30 20:57:49.039723 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.039681 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-credential-socket\") pod \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " Oct 30 20:57:49.039802 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.039748 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istiod-ca-cert" pod "2d0f604c-f9f8-40fb-ab5e-8517520166a3" (UID: "2d0f604c-f9f8-40fb-ab5e-8517520166a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:49.039917 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.039895 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-data\") pod \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " Oct 30 20:57:49.039975 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.039957 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-workload-socket\") pod \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " Oct 30 20:57:49.040041 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.040013 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-workload-certs\") pod \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\" (UID: \"2d0f604c-f9f8-40fb-ab5e-8517520166a3\") " Oct 30 20:57:49.041034 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.040558 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-workload-certs" pod "2d0f604c-f9f8-40fb-ab5e-8517520166a3" (UID: "2d0f604c-f9f8-40fb-ab5e-8517520166a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:49.041034 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.040568 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-workload-socket" pod "2d0f604c-f9f8-40fb-ab5e-8517520166a3" (UID: "2d0f604c-f9f8-40fb-ab5e-8517520166a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:49.041034 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.040587 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-data" pod "2d0f604c-f9f8-40fb-ab5e-8517520166a3" (UID: "2d0f604c-f9f8-40fb-ab5e-8517520166a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:49.041034 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.040783 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-credential-socket" pod "2d0f604c-f9f8-40fb-ab5e-8517520166a3" (UID: "2d0f604c-f9f8-40fb-ab5e-8517520166a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:49.041034 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.040822 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:49.041034 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.040837 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:49.041034 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.040853 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:49.041034 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.040868 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:49.041968 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.041943 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-token" pod "2d0f604c-f9f8-40fb-ab5e-8517520166a3" (UID: "2d0f604c-f9f8-40fb-ab5e-8517520166a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:49.042057 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.042013 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-envoy" pod "2d0f604c-f9f8-40fb-ab5e-8517520166a3" (UID: "2d0f604c-f9f8-40fb-ab5e-8517520166a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:49.042537 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.042515 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0f604c-f9f8-40fb-ab5e-8517520166a3-kube-api-access-vg6cp" pod "2d0f604c-f9f8-40fb-ab5e-8517520166a3" (UID: "2d0f604c-f9f8-40fb-ab5e-8517520166a3"). InnerVolumeSpecName "kube-api-access-vg6cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:49.043441 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.043421 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-podinfo" pod "2d0f604c-f9f8-40fb-ab5e-8517520166a3" (UID: "2d0f604c-f9f8-40fb-ab5e-8517520166a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:57:49.069395 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.069371 4848 scope.go:122] "RemoveContainer" containerID="0535ab83e33041130a6aa473ff5cdaf52048834874a057561e502aff896917ba" Oct 30 20:57:49.069805 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:57:49.069761 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0535ab83e33041130a6aa473ff5cdaf52048834874a057561e502aff896917ba\": container with ID starting with 0535ab83e33041130a6aa473ff5cdaf52048834874a057561e502aff896917ba not found: ID does not exist" containerID="0535ab83e33041130a6aa473ff5cdaf52048834874a057561e502aff896917ba" Oct 30 20:57:49.069918 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.069819 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0535ab83e33041130a6aa473ff5cdaf52048834874a057561e502aff896917ba"} err="failed to get container status \"0535ab83e33041130a6aa473ff5cdaf52048834874a057561e502aff896917ba\": rpc error: code = NotFound desc = could not find container \"0535ab83e33041130a6aa473ff5cdaf52048834874a057561e502aff896917ba\": container with ID starting with 0535ab83e33041130a6aa473ff5cdaf52048834874a057561e502aff896917ba not found: ID does not exist" Oct 30 20:57:49.069918 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.069843 4848 scope.go:122] "RemoveContainer" containerID="8c4aa4c4e8cf753eeceea15dea7b0bc657490cd1de2b382fa629f05006d908b1" Oct 30 20:57:49.070186 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:57:49.070165 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c4aa4c4e8cf753eeceea15dea7b0bc657490cd1de2b382fa629f05006d908b1\": container with ID starting with 8c4aa4c4e8cf753eeceea15dea7b0bc657490cd1de2b382fa629f05006d908b1 not found: ID does not exist" containerID="8c4aa4c4e8cf753eeceea15dea7b0bc657490cd1de2b382fa629f05006d908b1" Oct 30 20:57:49.070246 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.070229 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c4aa4c4e8cf753eeceea15dea7b0bc657490cd1de2b382fa629f05006d908b1"} err="failed to get container status \"8c4aa4c4e8cf753eeceea15dea7b0bc657490cd1de2b382fa629f05006d908b1\": rpc error: code = NotFound desc = could not find container \"8c4aa4c4e8cf753eeceea15dea7b0bc657490cd1de2b382fa629f05006d908b1\": container with ID starting with 8c4aa4c4e8cf753eeceea15dea7b0bc657490cd1de2b382fa629f05006d908b1 not found: ID does not exist" Oct 30 20:57:49.142239 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.142207 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:49.142239 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.142236 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:49.142434 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.142247 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg6cp\" (UniqueName: \"kubernetes.io/projected/2d0f604c-f9f8-40fb-ab5e-8517520166a3-kube-api-access-vg6cp\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:49.142434 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.142258 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d0f604c-f9f8-40fb-ab5e-8517520166a3-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:49.142434 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.142266 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d0f604c-f9f8-40fb-ab5e-8517520166a3-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:49.362915 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.362882 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx"] Oct 30 20:57:49.369735 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:49.369710 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c5bf9a-openshift-default-565dcfdd97-8srpx"] Oct 30 20:57:50.499894 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.499851 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz"] Oct 30 20:57:50.500591 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.500574 4848 cpu_manager.go:414] "containerMap: removing container" podUID="0d25fd62-4190-4453-bf50-38d631d53cf1" containerName="payload-pre-processing" Oct 30 20:57:50.500640 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.500593 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d25fd62-4190-4453-bf50-38d631d53cf1" containerName="payload-pre-processing" Oct 30 20:57:50.500640 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.500618 4848 cpu_manager.go:414] "containerMap: removing container" podUID="9ca3d842-f891-4981-a687-6971c181b6b7" containerName="payload-processing" Oct 30 20:57:50.500640 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.500624 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca3d842-f891-4981-a687-6971c181b6b7" containerName="payload-processing" Oct 30 20:57:50.500640 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.500636 4848 cpu_manager.go:414] "containerMap: removing container" podUID="86f1245b-8fd3-4831-abbd-fd9a428ce28b" containerName="istio-proxy" Oct 30 20:57:50.500942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.500641 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86f1245b-8fd3-4831-abbd-fd9a428ce28b" containerName="istio-proxy" Oct 30 20:57:50.500942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.500652 4848 cpu_manager.go:414] "containerMap: removing container" podUID="86f1245b-8fd3-4831-abbd-fd9a428ce28b" containerName="istio-proxy" Oct 30 20:57:50.500942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.500657 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86f1245b-8fd3-4831-abbd-fd9a428ce28b" containerName="istio-proxy" Oct 30 20:57:50.500942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.500680 4848 cpu_manager.go:414] "containerMap: removing container" podUID="2d0f604c-f9f8-40fb-ab5e-8517520166a3" containerName="istio-proxy" Oct 30 20:57:50.500942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.500686 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d0f604c-f9f8-40fb-ab5e-8517520166a3" containerName="istio-proxy" Oct 30 20:57:50.500942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.500782 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca3d842-f891-4981-a687-6971c181b6b7" containerName="payload-processing" Oct 30 20:57:50.500942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.500791 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="86f1245b-8fd3-4831-abbd-fd9a428ce28b" containerName="istio-proxy" Oct 30 20:57:50.500942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.500803 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="86f1245b-8fd3-4831-abbd-fd9a428ce28b" containerName="istio-proxy" Oct 30 20:57:50.500942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.500815 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d0f604c-f9f8-40fb-ab5e-8517520166a3" containerName="istio-proxy" Oct 30 20:57:50.500942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.500821 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d0f604c-f9f8-40fb-ab5e-8517520166a3" containerName="istio-proxy" Oct 30 20:57:50.500942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.500829 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d25fd62-4190-4453-bf50-38d631d53cf1" containerName="payload-pre-processing" Oct 30 20:57:50.504269 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.504250 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz" Oct 30 20:57:50.511880 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.511712 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-2b7ca7\"" Oct 30 20:57:50.517858 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.517830 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8"] Oct 30 20:57:50.518711 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.518691 4848 cpu_manager.go:414] "containerMap: removing container" podUID="2d0f604c-f9f8-40fb-ab5e-8517520166a3" containerName="istio-proxy" Oct 30 20:57:50.518711 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.518712 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d0f604c-f9f8-40fb-ab5e-8517520166a3" containerName="istio-proxy" Oct 30 20:57:50.522759 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.522739 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz"] Oct 30 20:57:50.522878 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.522847 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8" Oct 30 20:57:50.553746 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.553718 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-2b7ca7-dockercfg-kbpvd\"" Oct 30 20:57:50.558230 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.558202 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8"] Oct 30 20:57:50.564691 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.564666 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c26w\" (UniqueName: \"kubernetes.io/projected/1a12c1de-d86f-4748-93e9-b2729d60dbfd-kube-api-access-9c26w\") pod \"payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz\" (UID: \"1a12c1de-d86f-4748-93e9-b2729d60dbfd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz" Oct 30 20:57:50.564866 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.564848 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv7t8\" (UniqueName: \"kubernetes.io/projected/82df227a-4155-4961-be3b-a569fef21692-kube-api-access-hv7t8\") pod \"payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8\" (UID: \"82df227a-4155-4961-be3b-a569fef21692\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8" Oct 30 20:57:50.564983 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.564968 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a12c1de-d86f-4748-93e9-b2729d60dbfd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz\" (UID: \"1a12c1de-d86f-4748-93e9-b2729d60dbfd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz" Oct 30 20:57:50.565051 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.565033 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82df227a-4155-4961-be3b-a569fef21692-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8\" (UID: \"82df227a-4155-4961-be3b-a569fef21692\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8" Oct 30 20:57:50.681790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.678549 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a12c1de-d86f-4748-93e9-b2729d60dbfd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz\" (UID: \"1a12c1de-d86f-4748-93e9-b2729d60dbfd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz" Oct 30 20:57:50.681790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.678676 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82df227a-4155-4961-be3b-a569fef21692-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8\" (UID: \"82df227a-4155-4961-be3b-a569fef21692\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8" Oct 30 20:57:50.681790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.678969 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c26w\" (UniqueName: \"kubernetes.io/projected/1a12c1de-d86f-4748-93e9-b2729d60dbfd-kube-api-access-9c26w\") pod \"payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz\" (UID: \"1a12c1de-d86f-4748-93e9-b2729d60dbfd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz" Oct 30 20:57:50.681790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.679170 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv7t8\" (UniqueName: \"kubernetes.io/projected/82df227a-4155-4961-be3b-a569fef21692-kube-api-access-hv7t8\") pod \"payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8\" (UID: \"82df227a-4155-4961-be3b-a569fef21692\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8" Oct 30 20:57:50.681790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.680885 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a12c1de-d86f-4748-93e9-b2729d60dbfd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz\" (UID: \"1a12c1de-d86f-4748-93e9-b2729d60dbfd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz" Oct 30 20:57:50.681790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.681705 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82df227a-4155-4961-be3b-a569fef21692-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8\" (UID: \"82df227a-4155-4961-be3b-a569fef21692\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8" Oct 30 20:57:50.735844 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.726822 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv7t8\" (UniqueName: \"kubernetes.io/projected/82df227a-4155-4961-be3b-a569fef21692-kube-api-access-hv7t8\") pod \"payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8\" (UID: \"82df227a-4155-4961-be3b-a569fef21692\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8" Oct 30 20:57:50.739787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.737510 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c26w\" (UniqueName: \"kubernetes.io/projected/1a12c1de-d86f-4748-93e9-b2729d60dbfd-kube-api-access-9c26w\") pod \"payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz\" (UID: \"1a12c1de-d86f-4748-93e9-b2729d60dbfd\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz" Oct 30 20:57:50.821833 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.820651 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz" Oct 30 20:57:50.838369 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.838056 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8" Oct 30 20:57:50.983136 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:50.983099 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d0f604c-f9f8-40fb-ab5e-8517520166a3" path="/var/lib/kubelet/pods/2d0f604c-f9f8-40fb-ab5e-8517520166a3/volumes" Oct 30 20:57:51.242030 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:57:51.241999 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82df227a_4155_4961_be3b_a569fef21692.slice/crio-191376f85c81fbfc4681fb94846be143b0a45d5330de5d3c501820ec050eda3f WatchSource:0}: Error finding container 191376f85c81fbfc4681fb94846be143b0a45d5330de5d3c501820ec050eda3f: Status 404 returned error can't find the container with id 191376f85c81fbfc4681fb94846be143b0a45d5330de5d3c501820ec050eda3f Oct 30 20:57:51.242547 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:51.242512 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8"] Oct 30 20:57:51.242641 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:57:51.242584 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a12c1de_d86f_4748_93e9_b2729d60dbfd.slice/crio-bfb375bef79042bda32b93b6649c2c5803fb07866ce770a4817cc13c09a64cd4 WatchSource:0}: Error finding container bfb375bef79042bda32b93b6649c2c5803fb07866ce770a4817cc13c09a64cd4: Status 404 returned error can't find the container with id bfb375bef79042bda32b93b6649c2c5803fb07866ce770a4817cc13c09a64cd4 Oct 30 20:57:51.246040 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:51.246007 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz"] Oct 30 20:57:52.058529 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:52.058487 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8" event={"ID":"82df227a-4155-4961-be3b-a569fef21692","Type":"ContainerStarted","Data":"ae1b24ef2586df95b6a63cb6f39e0d90edc891adfd6be7d5f006bc3ca8d1b4f9"} Oct 30 20:57:52.059126 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:52.058560 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8" event={"ID":"82df227a-4155-4961-be3b-a569fef21692","Type":"ContainerStarted","Data":"191376f85c81fbfc4681fb94846be143b0a45d5330de5d3c501820ec050eda3f"} Oct 30 20:57:52.059126 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:52.058982 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8" Oct 30 20:57:52.068067 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:52.067972 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz" event={"ID":"1a12c1de-d86f-4748-93e9-b2729d60dbfd","Type":"ContainerStarted","Data":"75e2fc375131a69785f6d8a425b4c3d1a5a9e6247814942f1fb3cc79d1148706"} Oct 30 20:57:52.068231 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:52.068089 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz" event={"ID":"1a12c1de-d86f-4748-93e9-b2729d60dbfd","Type":"ContainerStarted","Data":"bfb375bef79042bda32b93b6649c2c5803fb07866ce770a4817cc13c09a64cd4"} Oct 30 20:57:52.083484 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:52.083418 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8" podStartSLOduration=2.083397026 podStartE2EDuration="2.083397026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:52.079388169 +0000 UTC m=+1177.259284677" watchObservedRunningTime="2026-10-30 20:57:52.083397026 +0000 UTC m=+1177.263293533" Oct 30 20:57:52.102601 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:52.102540 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz" podStartSLOduration=2.102519799 podStartE2EDuration="2.102519799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:52.09794452 +0000 UTC m=+1177.277841032" watchObservedRunningTime="2026-10-30 20:57:52.102519799 +0000 UTC m=+1177.282416308" Oct 30 20:57:52.619347 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:57:52.619300 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:57:53.074855 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:53.074825 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz" Oct 30 20:57:54.521014 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:54.520980 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn"] Oct 30 20:57:54.521507 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:54.521242 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" podUID="a53b4de2-a49e-44d8-bc2e-369e52309561" containerName="istio-proxy" containerID="cri-o://4a3c6f0f4c15e09a78281bd86a8dedaf4851c8ca9058caa6087e3370304aeb12" gracePeriod=30 Oct 30 20:57:56.420866 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.420830 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb"] Oct 30 20:57:56.426289 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.426267 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.435458 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.435429 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb"] Oct 30 20:57:56.437733 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.436803 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-38c5ca-openshift-default-dockercfg-8htr2\"" Oct 30 20:57:56.635751 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.633405 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-workload-certs\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.635751 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.633485 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-envoy\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.635751 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.633552 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-data\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.635751 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.633686 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-credential-socket\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.635751 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.633753 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-workload-socket\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.635751 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.633806 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-token\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.635751 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.633932 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gs52\" (UniqueName: \"kubernetes.io/projected/bed55d4a-5593-41db-84e1-fe96bb9a849b-kube-api-access-9gs52\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.635751 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.633959 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-podinfo\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.635751 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.633991 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bed55d4a-5593-41db-84e1-fe96bb9a849b-istiod-ca-cert\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.747791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.740063 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bed55d4a-5593-41db-84e1-fe96bb9a849b-istiod-ca-cert\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.747791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.742047 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-workload-certs\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.747791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.742078 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-envoy\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.747791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.742109 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-data\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.747791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.742216 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-credential-socket\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.747791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.742273 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-workload-socket\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.747791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.742305 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-token\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.747791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.742411 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gs52\" (UniqueName: \"kubernetes.io/projected/bed55d4a-5593-41db-84e1-fe96bb9a849b-kube-api-access-9gs52\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.747791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.742442 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-podinfo\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.765793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.763988 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-data\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.765793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.741068 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bed55d4a-5593-41db-84e1-fe96bb9a849b-istiod-ca-cert\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.765793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.764279 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-workload-certs\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.765793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.765365 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-credential-socket\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.766137 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.765892 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-workload-socket\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.794791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.794303 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-token\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.805788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.804003 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gs52\" (UniqueName: \"kubernetes.io/projected/bed55d4a-5593-41db-84e1-fe96bb9a849b-kube-api-access-9gs52\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.805788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.804287 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-envoy\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.824785 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.823545 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-podinfo\") pod \"e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:56.950076 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.950041 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d"] Oct 30 20:57:56.956642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.956613 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" Oct 30 20:57:56.979492 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:56.979465 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d"] Oct 30 20:57:57.006849 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.006750 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-2b7ca7-serving-cert\"" Oct 30 20:57:57.043953 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.043918 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:57.057890 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.057862 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d\" (UID: \"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" Oct 30 20:57:57.057999 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.057968 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-metrics-certs\") pod \"maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d\" (UID: \"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" Oct 30 20:57:57.058074 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.058020 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d\" (UID: \"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" Oct 30 20:57:57.058133 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.058072 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28m2w\" (UniqueName: \"kubernetes.io/projected/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-kube-api-access-28m2w\") pod \"maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d\" (UID: \"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" Oct 30 20:57:57.171564 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.170605 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d\" (UID: \"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" Oct 30 20:57:57.171564 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.170728 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-metrics-certs\") pod \"maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d\" (UID: \"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" Oct 30 20:57:57.171564 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.170813 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d\" (UID: \"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" Oct 30 20:57:57.171564 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.170897 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28m2w\" (UniqueName: \"kubernetes.io/projected/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-kube-api-access-28m2w\") pod \"maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d\" (UID: \"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" Oct 30 20:57:57.172103 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.172030 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d\" (UID: \"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" Oct 30 20:57:57.181451 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.180007 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-metrics-certs\") pod \"maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d\" (UID: \"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" Oct 30 20:57:57.184014 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.183317 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28m2w\" (UniqueName: \"kubernetes.io/projected/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-kube-api-access-28m2w\") pod \"maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d\" (UID: \"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" Oct 30 20:57:57.196860 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.196785 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d\" (UID: \"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" Oct 30 20:57:57.295253 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.294053 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" Oct 30 20:57:57.333935 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.333904 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb"] Oct 30 20:57:57.334740 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:57:57.334715 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed55d4a_5593_41db_84e1_fe96bb9a849b.slice/crio-ba75e919363db80a0f2577dc7238fa49511a558c155a18a8929f53d874c4925a WatchSource:0}: Error finding container ba75e919363db80a0f2577dc7238fa49511a558c155a18a8929f53d874c4925a: Status 404 returned error can't find the container with id ba75e919363db80a0f2577dc7238fa49511a558c155a18a8929f53d874c4925a Oct 30 20:57:57.337347 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.337315 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:57.337408 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.337385 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:57.337452 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.337413 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:57:57.608036 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.608006 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d"] Oct 30 20:57:57.608924 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:57:57.608902 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab4ad8f_4f9f_4b3c_a7d8_f8b0af3b5eb2.slice/crio-329d351e9142d7a7689ebfcc65a9fe9ff6de32be626471612cde2914c01fba07 WatchSource:0}: Error finding container 329d351e9142d7a7689ebfcc65a9fe9ff6de32be626471612cde2914c01fba07: Status 404 returned error can't find the container with id 329d351e9142d7a7689ebfcc65a9fe9ff6de32be626471612cde2914c01fba07 Oct 30 20:57:57.610959 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:57.610942 4848 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:57:58.162988 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:58.162943 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" event={"ID":"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2","Type":"ContainerStarted","Data":"329d351e9142d7a7689ebfcc65a9fe9ff6de32be626471612cde2914c01fba07"} Oct 30 20:57:58.165732 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:58.165698 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" event={"ID":"bed55d4a-5593-41db-84e1-fe96bb9a849b","Type":"ContainerStarted","Data":"d898fb74ee6342dd97b9e02eeb7b940d39ef5af38b64dc2b5c7fe62aaa4abdd6"} Oct 30 20:57:58.165964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:58.165788 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" event={"ID":"bed55d4a-5593-41db-84e1-fe96bb9a849b","Type":"ContainerStarted","Data":"ba75e919363db80a0f2577dc7238fa49511a558c155a18a8929f53d874c4925a"} Oct 30 20:57:58.189328 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:58.189266 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" podStartSLOduration=2.189250888 podStartE2EDuration="2.189250888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:58.187659597 +0000 UTC m=+1183.367556106" watchObservedRunningTime="2026-10-30 20:57:58.189250888 +0000 UTC m=+1183.369147395" Oct 30 20:57:59.044504 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.044464 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:59.050513 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.050482 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:59.174494 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.174461 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" event={"ID":"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2","Type":"ContainerStarted","Data":"718b4189c1379f6f2040ef44aa78ba99b7729dbaed5eb9e436e5f9a352974246"} Oct 30 20:57:59.174689 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.174606 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:59.174757 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.174739 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" Oct 30 20:57:59.175589 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.175573 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:57:59.200842 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.200761 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" podStartSLOduration=2.584080859 podStartE2EDuration="3.200746905s" totalImagesPullingTime="616.666046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:57:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:57:59.197816598 +0000 UTC m=+1184.377713107" watchObservedRunningTime="2026-10-30 20:57:59.200746905 +0000 UTC m=+1184.380643413" Oct 30 20:57:59.798321 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.798259 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:57:59.860938 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.860904 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-envoy\") pod \"a53b4de2-a49e-44d8-bc2e-369e52309561\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " Oct 30 20:57:59.861269 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.861238 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-workload-certs\") pod \"a53b4de2-a49e-44d8-bc2e-369e52309561\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " Oct 30 20:57:59.861430 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.861297 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53b4de2-a49e-44d8-bc2e-369e52309561-kube-api-access-6kvcs\" (UniqueName: \"kubernetes.io/projected/a53b4de2-a49e-44d8-bc2e-369e52309561-kube-api-access-6kvcs\") pod \"a53b4de2-a49e-44d8-bc2e-369e52309561\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " Oct 30 20:57:59.861534 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.861510 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-workload-certs" pod "a53b4de2-a49e-44d8-bc2e-369e52309561" (UID: "a53b4de2-a49e-44d8-bc2e-369e52309561"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:59.862136 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.861869 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-data\") pod \"a53b4de2-a49e-44d8-bc2e-369e52309561\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " Oct 30 20:57:59.862340 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.861963 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-podinfo\") pod \"a53b4de2-a49e-44d8-bc2e-369e52309561\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " Oct 30 20:57:59.862459 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.862115 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-data" pod "a53b4de2-a49e-44d8-bc2e-369e52309561" (UID: "a53b4de2-a49e-44d8-bc2e-369e52309561"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:59.862585 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.862465 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-token\" (UniqueName: \"kubernetes.io/projected/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-token\") pod \"a53b4de2-a49e-44d8-bc2e-369e52309561\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " Oct 30 20:57:59.863783 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.863185 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-workload-socket" pod "a53b4de2-a49e-44d8-bc2e-369e52309561" (UID: "a53b4de2-a49e-44d8-bc2e-369e52309561"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:59.863783 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.862858 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-workload-socket\") pod \"a53b4de2-a49e-44d8-bc2e-369e52309561\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " Oct 30 20:57:59.864886 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.864859 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53b4de2-a49e-44d8-bc2e-369e52309561-istiod-ca-cert" pod "a53b4de2-a49e-44d8-bc2e-369e52309561" (UID: "a53b4de2-a49e-44d8-bc2e-369e52309561"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:57:59.865516 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.865488 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53b4de2-a49e-44d8-bc2e-369e52309561-kube-api-access-6kvcs" pod "a53b4de2-a49e-44d8-bc2e-369e52309561" (UID: "a53b4de2-a49e-44d8-bc2e-369e52309561"). InnerVolumeSpecName "kube-api-access-6kvcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:59.865614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.865511 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-podinfo" pod "a53b4de2-a49e-44d8-bc2e-369e52309561" (UID: "a53b4de2-a49e-44d8-bc2e-369e52309561"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:57:59.865614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.863676 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53b4de2-a49e-44d8-bc2e-369e52309561-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53b4de2-a49e-44d8-bc2e-369e52309561-istiod-ca-cert\") pod \"a53b4de2-a49e-44d8-bc2e-369e52309561\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " Oct 30 20:57:59.865614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.865604 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-credential-socket\") pod \"a53b4de2-a49e-44d8-bc2e-369e52309561\" (UID: \"a53b4de2-a49e-44d8-bc2e-369e52309561\") " Oct 30 20:57:59.865966 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.865932 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-credential-socket" pod "a53b4de2-a49e-44d8-bc2e-369e52309561" (UID: "a53b4de2-a49e-44d8-bc2e-369e52309561"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:59.866253 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.866222 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-envoy" pod "a53b4de2-a49e-44d8-bc2e-369e52309561" (UID: "a53b4de2-a49e-44d8-bc2e-369e52309561"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:57:59.866253 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.866243 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-token" pod "a53b4de2-a49e-44d8-bc2e-369e52309561" (UID: "a53b4de2-a49e-44d8-bc2e-369e52309561"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:57:59.866514 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.866500 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:59.866584 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.866520 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:59.866584 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.866534 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:59.866584 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.866548 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:59.866584 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.866561 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a53b4de2-a49e-44d8-bc2e-369e52309561-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:59.866584 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.866574 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:59.866897 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.866587 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:59.866897 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.866601 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a53b4de2-a49e-44d8-bc2e-369e52309561-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:57:59.866897 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:57:59.866614 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kvcs\" (UniqueName: \"kubernetes.io/projected/a53b4de2-a49e-44d8-bc2e-369e52309561-kube-api-access-6kvcs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:00.183404 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:00.183374 4848 generic.go:356] "Generic (PLEG): container finished" podID="a53b4de2-a49e-44d8-bc2e-369e52309561" containerID="4a3c6f0f4c15e09a78281bd86a8dedaf4851c8ca9058caa6087e3370304aeb12" exitCode=0 Oct 30 20:58:00.183866 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:00.183431 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" Oct 30 20:58:00.183866 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:00.183467 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" event={"ID":"a53b4de2-a49e-44d8-bc2e-369e52309561","Type":"ContainerDied","Data":"4a3c6f0f4c15e09a78281bd86a8dedaf4851c8ca9058caa6087e3370304aeb12"} Oct 30 20:58:00.183866 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:00.183519 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn" event={"ID":"a53b4de2-a49e-44d8-bc2e-369e52309561","Type":"ContainerDied","Data":"5bbc03253a3d624d5dfd553d2fabd5b0df0494621a53c16c6f9143316b7cc4fe"} Oct 30 20:58:00.183866 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:00.183546 4848 scope.go:122] "RemoveContainer" containerID="4a3c6f0f4c15e09a78281bd86a8dedaf4851c8ca9058caa6087e3370304aeb12" Oct 30 20:58:00.199565 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:00.199545 4848 scope.go:122] "RemoveContainer" containerID="de629f83777a61eb2754c58623a1973ca282be13d91288429c1233a95c42a060" Oct 30 20:58:00.209263 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:00.209232 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn"] Oct 30 20:58:00.210447 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:00.210423 4848 scope.go:122] "RemoveContainer" containerID="4a3c6f0f4c15e09a78281bd86a8dedaf4851c8ca9058caa6087e3370304aeb12" Oct 30 20:58:00.210876 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:58:00.210854 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3c6f0f4c15e09a78281bd86a8dedaf4851c8ca9058caa6087e3370304aeb12\": container with ID starting with 4a3c6f0f4c15e09a78281bd86a8dedaf4851c8ca9058caa6087e3370304aeb12 not found: ID does not exist" containerID="4a3c6f0f4c15e09a78281bd86a8dedaf4851c8ca9058caa6087e3370304aeb12" Oct 30 20:58:00.210974 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:00.210883 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3c6f0f4c15e09a78281bd86a8dedaf4851c8ca9058caa6087e3370304aeb12"} err="failed to get container status \"4a3c6f0f4c15e09a78281bd86a8dedaf4851c8ca9058caa6087e3370304aeb12\": rpc error: code = NotFound desc = could not find container \"4a3c6f0f4c15e09a78281bd86a8dedaf4851c8ca9058caa6087e3370304aeb12\": container with ID starting with 4a3c6f0f4c15e09a78281bd86a8dedaf4851c8ca9058caa6087e3370304aeb12 not found: ID does not exist" Oct 30 20:58:00.210974 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:00.210906 4848 scope.go:122] "RemoveContainer" containerID="de629f83777a61eb2754c58623a1973ca282be13d91288429c1233a95c42a060" Oct 30 20:58:00.211216 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:58:00.211200 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de629f83777a61eb2754c58623a1973ca282be13d91288429c1233a95c42a060\": container with ID starting with de629f83777a61eb2754c58623a1973ca282be13d91288429c1233a95c42a060 not found: ID does not exist" containerID="de629f83777a61eb2754c58623a1973ca282be13d91288429c1233a95c42a060" Oct 30 20:58:00.211277 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:00.211220 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de629f83777a61eb2754c58623a1973ca282be13d91288429c1233a95c42a060"} err="failed to get container status \"de629f83777a61eb2754c58623a1973ca282be13d91288429c1233a95c42a060\": rpc error: code = NotFound desc = could not find container \"de629f83777a61eb2754c58623a1973ca282be13d91288429c1233a95c42a060\": container with ID starting with de629f83777a61eb2754c58623a1973ca282be13d91288429c1233a95c42a060 not found: ID does not exist" Oct 30 20:58:00.212816 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:00.212793 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a61855-openshift-default-8596696d9c-dv7rn"] Oct 30 20:58:00.933657 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:00.933627 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a53b4de2-a49e-44d8-bc2e-369e52309561" path="/var/lib/kubelet/pods/a53b4de2-a49e-44d8-bc2e-369e52309561/volumes" Oct 30 20:58:01.880189 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:01.880152 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b"] Oct 30 20:58:01.880794 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:01.880780 4848 cpu_manager.go:414] "containerMap: removing container" podUID="a53b4de2-a49e-44d8-bc2e-369e52309561" containerName="istio-proxy" Oct 30 20:58:01.880846 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:01.880794 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53b4de2-a49e-44d8-bc2e-369e52309561" containerName="istio-proxy" Oct 30 20:58:01.880846 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:01.880811 4848 cpu_manager.go:414] "containerMap: removing container" podUID="a53b4de2-a49e-44d8-bc2e-369e52309561" containerName="istio-proxy" Oct 30 20:58:01.880846 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:01.880817 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53b4de2-a49e-44d8-bc2e-369e52309561" containerName="istio-proxy" Oct 30 20:58:01.880955 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:01.880945 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53b4de2-a49e-44d8-bc2e-369e52309561" containerName="istio-proxy" Oct 30 20:58:01.880991 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:01.880958 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53b4de2-a49e-44d8-bc2e-369e52309561" containerName="istio-proxy" Oct 30 20:58:01.885937 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:01.885912 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b" Oct 30 20:58:01.895496 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:01.894631 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-38c5ca\"" Oct 30 20:58:01.897622 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:01.897596 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-38c5ca-dockercfg-k9g87\"" Oct 30 20:58:01.900664 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:01.900636 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b"] Oct 30 20:58:01.926960 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:01.926920 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3101d0-f69c-47ef-b438-6fc1966e913b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b\" (UID: \"6b3101d0-f69c-47ef-b438-6fc1966e913b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b" Oct 30 20:58:01.927176 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:01.927150 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96jqg\" (UniqueName: \"kubernetes.io/projected/6b3101d0-f69c-47ef-b438-6fc1966e913b-kube-api-access-96jqg\") pod \"payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b\" (UID: \"6b3101d0-f69c-47ef-b438-6fc1966e913b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b" Oct 30 20:58:01.943343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:01.943301 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h"] Oct 30 20:58:01.948970 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:01.948942 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h" Oct 30 20:58:01.960487 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:01.960455 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h"] Oct 30 20:58:02.036790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:02.030795 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3101d0-f69c-47ef-b438-6fc1966e913b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b\" (UID: \"6b3101d0-f69c-47ef-b438-6fc1966e913b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b" Oct 30 20:58:02.036790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:02.030880 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1447449b-6f3f-4cde-9a36-0f56ce7a8b16-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h\" (UID: \"1447449b-6f3f-4cde-9a36-0f56ce7a8b16\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h" Oct 30 20:58:02.036790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:02.030943 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-582bl\" (UniqueName: \"kubernetes.io/projected/1447449b-6f3f-4cde-9a36-0f56ce7a8b16-kube-api-access-582bl\") pod \"payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h\" (UID: \"1447449b-6f3f-4cde-9a36-0f56ce7a8b16\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h" Oct 30 20:58:02.036790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:02.031059 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96jqg\" (UniqueName: \"kubernetes.io/projected/6b3101d0-f69c-47ef-b438-6fc1966e913b-kube-api-access-96jqg\") pod \"payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b\" (UID: \"6b3101d0-f69c-47ef-b438-6fc1966e913b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b" Oct 30 20:58:02.036790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:02.032132 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3101d0-f69c-47ef-b438-6fc1966e913b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b\" (UID: \"6b3101d0-f69c-47ef-b438-6fc1966e913b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b" Oct 30 20:58:02.148895 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:02.147947 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1447449b-6f3f-4cde-9a36-0f56ce7a8b16-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h\" (UID: \"1447449b-6f3f-4cde-9a36-0f56ce7a8b16\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h" Oct 30 20:58:02.148895 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:02.148032 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-582bl\" (UniqueName: \"kubernetes.io/projected/1447449b-6f3f-4cde-9a36-0f56ce7a8b16-kube-api-access-582bl\") pod \"payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h\" (UID: \"1447449b-6f3f-4cde-9a36-0f56ce7a8b16\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h" Oct 30 20:58:02.180789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:02.149281 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1447449b-6f3f-4cde-9a36-0f56ce7a8b16-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h\" (UID: \"1447449b-6f3f-4cde-9a36-0f56ce7a8b16\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h" Oct 30 20:58:02.231461 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:02.223655 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96jqg\" (UniqueName: \"kubernetes.io/projected/6b3101d0-f69c-47ef-b438-6fc1966e913b-kube-api-access-96jqg\") pod \"payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b\" (UID: \"6b3101d0-f69c-47ef-b438-6fc1966e913b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b" Oct 30 20:58:02.253512 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:02.237138 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b" Oct 30 20:58:02.282611 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:02.278371 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-582bl\" (UniqueName: \"kubernetes.io/projected/1447449b-6f3f-4cde-9a36-0f56ce7a8b16-kube-api-access-582bl\") pod \"payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h\" (UID: \"1447449b-6f3f-4cde-9a36-0f56ce7a8b16\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h" Oct 30 20:58:02.568744 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:02.568710 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h" Oct 30 20:58:02.607928 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:02.607891 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b"] Oct 30 20:58:02.612807 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:58:02.612749 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3101d0_f69c_47ef_b438_6fc1966e913b.slice/crio-c451e946eba18a3b90b6f1975277a9b189e9cab0d110a2ea29bfbc71eba37d25 WatchSource:0}: Error finding container c451e946eba18a3b90b6f1975277a9b189e9cab0d110a2ea29bfbc71eba37d25: Status 404 returned error can't find the container with id c451e946eba18a3b90b6f1975277a9b189e9cab0d110a2ea29bfbc71eba37d25 Oct 30 20:58:02.859497 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:02.859466 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h"] Oct 30 20:58:02.860362 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:58:02.860327 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1447449b_6f3f_4cde_9a36_0f56ce7a8b16.slice/crio-6073d3e727c27f29a14d9d1c27ee437160b14d3e0eb3d56e8bfa8e06ba6ef9f3 WatchSource:0}: Error finding container 6073d3e727c27f29a14d9d1c27ee437160b14d3e0eb3d56e8bfa8e06ba6ef9f3: Status 404 returned error can't find the container with id 6073d3e727c27f29a14d9d1c27ee437160b14d3e0eb3d56e8bfa8e06ba6ef9f3 Oct 30 20:58:03.077500 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:03.077471 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8" Oct 30 20:58:03.323434 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:03.323400 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b" event={"ID":"6b3101d0-f69c-47ef-b438-6fc1966e913b","Type":"ContainerStarted","Data":"9ca4d934c73bfc206688163a956f6a6f4cc535dd5781df899bb168cb33372915"} Oct 30 20:58:03.323665 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:03.323462 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b" event={"ID":"6b3101d0-f69c-47ef-b438-6fc1966e913b","Type":"ContainerStarted","Data":"c451e946eba18a3b90b6f1975277a9b189e9cab0d110a2ea29bfbc71eba37d25"} Oct 30 20:58:03.323665 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:03.323602 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b" Oct 30 20:58:03.325346 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:03.325322 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h" event={"ID":"1447449b-6f3f-4cde-9a36-0f56ce7a8b16","Type":"ContainerStarted","Data":"96a726b3b21406d49105f334a20dd32f89fad19a85b11b87a9de333abe75cd5c"} Oct 30 20:58:03.325462 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:03.325373 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h" Oct 30 20:58:03.325462 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:03.325396 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h" event={"ID":"1447449b-6f3f-4cde-9a36-0f56ce7a8b16","Type":"ContainerStarted","Data":"6073d3e727c27f29a14d9d1c27ee437160b14d3e0eb3d56e8bfa8e06ba6ef9f3"} Oct 30 20:58:03.340395 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:03.340322 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b" podStartSLOduration=2.340307235 podStartE2EDuration="2.340307235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:03.339016839 +0000 UTC m=+1188.518913347" watchObservedRunningTime="2026-10-30 20:58:03.340307235 +0000 UTC m=+1188.520203743" Oct 30 20:58:03.357864 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:03.357808 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h" podStartSLOduration=2.357790909 podStartE2EDuration="2.357790909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:03.354152464 +0000 UTC m=+1188.534048961" watchObservedRunningTime="2026-10-30 20:58:03.357790909 +0000 UTC m=+1188.537687415" Oct 30 20:58:04.101456 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:04.101428 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz" Oct 30 20:58:05.190182 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:05.190152 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" Oct 30 20:58:05.806883 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:05.806851 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws"] Oct 30 20:58:05.807309 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:05.807283 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" podUID="0c4e64ec-eb49-4a9e-86df-7483d597fd9c" containerName="istio-proxy" containerID="cri-o://f3a46c24f7495f934a072dd252625430b6bde5607733410e883217c324c7e64f" gracePeriod=30 Oct 30 20:58:08.099663 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:08.098005 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:58:08.099663 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:08.098087 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" podUID="0c4e64ec-eb49-4a9e-86df-7483d597fd9c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:58:09.792869 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:09.792833 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs"] Oct 30 20:58:09.793366 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:09.793141 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" podUID="883d31e2-ac6d-466c-900d-d86010b4028e" containerName="istio-proxy" containerID="cri-o://d96dd712b792441d9bd648e9df77a084aecdf2f904edcc3b3eadc340f3c64956" gracePeriod=30 Oct 30 20:58:09.798261 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:09.798236 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m"] Oct 30 20:58:09.798487 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:09.798462 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" podUID="16e788c7-69cc-48cb-959a-62c4763c2698" containerName="istio-proxy" containerID="cri-o://97484869dee74327eaf742fa17348a1abddd4ca77456e3d2017caa1046a662fd" gracePeriod=30 Oct 30 20:58:09.801340 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:09.801314 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx"] Oct 30 20:58:09.801524 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:09.801503 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" podUID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" containerName="istio-proxy" containerID="cri-o://1fee94e5820b62c606edde78e48dcedf152c59ecae177ace9c6929c5c40509a8" gracePeriod=30 Oct 30 20:58:09.884189 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:09.884154 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:58:09.884385 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:09.884228 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" podUID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:58:11.186807 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.184068 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:58:11.268803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.262597 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-workload-certs\") pod \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " Oct 30 20:58:11.268803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.262703 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istiod-ca-cert\") pod \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " Oct 30 20:58:11.268803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.262848 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-podinfo\") pod \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " Oct 30 20:58:11.268803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.263600 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-workload-certs" pod "0c4e64ec-eb49-4a9e-86df-7483d597fd9c" (UID: "0c4e64ec-eb49-4a9e-86df-7483d597fd9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:11.268803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.263964 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istiod-ca-cert" pod "0c4e64ec-eb49-4a9e-86df-7483d597fd9c" (UID: "0c4e64ec-eb49-4a9e-86df-7483d597fd9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:11.268803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.264023 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-data\") pod \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " Oct 30 20:58:11.268803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.264085 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-envoy\") pod \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " Oct 30 20:58:11.268803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.264126 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-kube-api-access-zkz59\" (UniqueName: \"kubernetes.io/projected/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-kube-api-access-zkz59\") pod \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " Oct 30 20:58:11.268803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.264190 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-token\" (UniqueName: \"kubernetes.io/projected/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-token\") pod \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " Oct 30 20:58:11.268803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.264221 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-workload-socket\") pod \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " Oct 30 20:58:11.268803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.264340 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-credential-socket\") pod \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\" (UID: \"0c4e64ec-eb49-4a9e-86df-7483d597fd9c\") " Oct 30 20:58:11.268803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.265085 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:11.268803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.265107 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:11.281454 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.280906 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-podinfo" pod "0c4e64ec-eb49-4a9e-86df-7483d597fd9c" (UID: "0c4e64ec-eb49-4a9e-86df-7483d597fd9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:11.304395 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.303607 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-workload-socket" pod "0c4e64ec-eb49-4a9e-86df-7483d597fd9c" (UID: "0c4e64ec-eb49-4a9e-86df-7483d597fd9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:11.304395 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.303854 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-data" pod "0c4e64ec-eb49-4a9e-86df-7483d597fd9c" (UID: "0c4e64ec-eb49-4a9e-86df-7483d597fd9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:11.352875 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.340947 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-credential-socket" pod "0c4e64ec-eb49-4a9e-86df-7483d597fd9c" (UID: "0c4e64ec-eb49-4a9e-86df-7483d597fd9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:11.371791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.359544 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-token" pod "0c4e64ec-eb49-4a9e-86df-7483d597fd9c" (UID: "0c4e64ec-eb49-4a9e-86df-7483d597fd9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:11.371791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.361021 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-kube-api-access-zkz59" pod "0c4e64ec-eb49-4a9e-86df-7483d597fd9c" (UID: "0c4e64ec-eb49-4a9e-86df-7483d597fd9c"). InnerVolumeSpecName "kube-api-access-zkz59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:11.371791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.366590 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkz59\" (UniqueName: \"kubernetes.io/projected/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-kube-api-access-zkz59\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:11.371791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.366622 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:11.371791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.366638 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:11.371791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.366656 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:11.371791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.366671 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:11.371791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.366686 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:11.371791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.371038 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-envoy" pod "0c4e64ec-eb49-4a9e-86df-7483d597fd9c" (UID: "0c4e64ec-eb49-4a9e-86df-7483d597fd9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:11.400947 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.397814 4848 generic.go:356] "Generic (PLEG): container finished" podID="0c4e64ec-eb49-4a9e-86df-7483d597fd9c" containerID="f3a46c24f7495f934a072dd252625430b6bde5607733410e883217c324c7e64f" exitCode=0 Oct 30 20:58:11.400947 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.397886 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" Oct 30 20:58:11.400947 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.397905 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" event={"ID":"0c4e64ec-eb49-4a9e-86df-7483d597fd9c","Type":"ContainerDied","Data":"f3a46c24f7495f934a072dd252625430b6bde5607733410e883217c324c7e64f"} Oct 30 20:58:11.400947 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.397972 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws" event={"ID":"0c4e64ec-eb49-4a9e-86df-7483d597fd9c","Type":"ContainerDied","Data":"e788e737f30b2178c9a6622d638db0f8a4db8d6fc7c2b49332dd770ceb564284"} Oct 30 20:58:11.400947 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.398008 4848 scope.go:122] "RemoveContainer" containerID="f3a46c24f7495f934a072dd252625430b6bde5607733410e883217c324c7e64f" Oct 30 20:58:11.429593 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.429561 4848 scope.go:122] "RemoveContainer" containerID="95b6dc03d99fc0248e0083fa8cd12911f66e177064eb0980fd6b9ba309ba5fcc" Oct 30 20:58:11.446049 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.446015 4848 scope.go:122] "RemoveContainer" containerID="f3a46c24f7495f934a072dd252625430b6bde5607733410e883217c324c7e64f" Oct 30 20:58:11.446511 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:58:11.446483 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a46c24f7495f934a072dd252625430b6bde5607733410e883217c324c7e64f\": container with ID starting with f3a46c24f7495f934a072dd252625430b6bde5607733410e883217c324c7e64f not found: ID does not exist" containerID="f3a46c24f7495f934a072dd252625430b6bde5607733410e883217c324c7e64f" Oct 30 20:58:11.446602 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.446525 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a46c24f7495f934a072dd252625430b6bde5607733410e883217c324c7e64f"} err="failed to get container status \"f3a46c24f7495f934a072dd252625430b6bde5607733410e883217c324c7e64f\": rpc error: code = NotFound desc = could not find container \"f3a46c24f7495f934a072dd252625430b6bde5607733410e883217c324c7e64f\": container with ID starting with f3a46c24f7495f934a072dd252625430b6bde5607733410e883217c324c7e64f not found: ID does not exist" Oct 30 20:58:11.446602 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.446555 4848 scope.go:122] "RemoveContainer" containerID="95b6dc03d99fc0248e0083fa8cd12911f66e177064eb0980fd6b9ba309ba5fcc" Oct 30 20:58:11.446931 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:58:11.446905 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b6dc03d99fc0248e0083fa8cd12911f66e177064eb0980fd6b9ba309ba5fcc\": container with ID starting with 95b6dc03d99fc0248e0083fa8cd12911f66e177064eb0980fd6b9ba309ba5fcc not found: ID does not exist" containerID="95b6dc03d99fc0248e0083fa8cd12911f66e177064eb0980fd6b9ba309ba5fcc" Oct 30 20:58:11.447007 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.446964 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b6dc03d99fc0248e0083fa8cd12911f66e177064eb0980fd6b9ba309ba5fcc"} err="failed to get container status \"95b6dc03d99fc0248e0083fa8cd12911f66e177064eb0980fd6b9ba309ba5fcc\": rpc error: code = NotFound desc = could not find container \"95b6dc03d99fc0248e0083fa8cd12911f66e177064eb0980fd6b9ba309ba5fcc\": container with ID starting with 95b6dc03d99fc0248e0083fa8cd12911f66e177064eb0980fd6b9ba309ba5fcc not found: ID does not exist" Oct 30 20:58:11.453040 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.453005 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws"] Oct 30 20:58:11.465111 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.465073 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-351fd1-openshift-default-74f5c98f57-7vsws"] Oct 30 20:58:11.469130 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:11.469092 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c4e64ec-eb49-4a9e-86df-7483d597fd9c-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:12.081705 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.081608 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5"] Oct 30 20:58:12.096105 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.096061 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" podUID="96a95230-aa96-4d6b-856e-e29871f3d3bf" containerName="istio-proxy" containerID="cri-o://ac1fdb670c5c8dfb173984560945b31220d115e913cc25d7cbcc23a47b413f15" gracePeriod=30 Oct 30 20:58:12.334280 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.334248 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx"] Oct 30 20:58:12.335133 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.335097 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" podUID="794057dc-d92a-4c15-8003-d3e8132297da" containerName="istio-proxy" containerID="cri-o://1f64d311a40d18931f63c5037f1375d1ea0e552d26dc166786963a74d0e63c74" gracePeriod=30 Oct 30 20:58:12.353827 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.343744 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb"] Oct 30 20:58:12.353827 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.343993 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" podUID="f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" containerName="istio-proxy" containerID="cri-o://a20a378097eceae6fc8987bbc2b885b752fb17ef981f1359169809adfbcf8bbc" gracePeriod=30 Oct 30 20:58:12.842290 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.842237 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh"] Oct 30 20:58:12.843133 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.843108 4848 cpu_manager.go:414] "containerMap: removing container" podUID="0c4e64ec-eb49-4a9e-86df-7483d597fd9c" containerName="istio-proxy" Oct 30 20:58:12.843278 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.843263 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c4e64ec-eb49-4a9e-86df-7483d597fd9c" containerName="istio-proxy" Oct 30 20:58:12.843364 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.843355 4848 cpu_manager.go:414] "containerMap: removing container" podUID="0c4e64ec-eb49-4a9e-86df-7483d597fd9c" containerName="istio-proxy" Oct 30 20:58:12.843430 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.843422 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c4e64ec-eb49-4a9e-86df-7483d597fd9c" containerName="istio-proxy" Oct 30 20:58:12.843609 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.843596 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c4e64ec-eb49-4a9e-86df-7483d597fd9c" containerName="istio-proxy" Oct 30 20:58:12.853157 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.853131 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh" Oct 30 20:58:12.853937 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.853908 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh"] Oct 30 20:58:12.857932 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.857895 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-c72dc9\"" Oct 30 20:58:12.858113 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.858056 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-c72dc9-dockercfg-8k7dd\"" Oct 30 20:58:12.894309 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.894270 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz"] Oct 30 20:58:12.899837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.897410 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c4e64ec-eb49-4a9e-86df-7483d597fd9c" containerName="istio-proxy" Oct 30 20:58:12.904240 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.902837 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz" Oct 30 20:58:12.959163 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.959122 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac51789-e545-4331-ae29-f99e8150fc15-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh\" (UID: \"9ac51789-e545-4331-ae29-f99e8150fc15\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh" Oct 30 20:58:12.959421 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.959202 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5pvx\" (UniqueName: \"kubernetes.io/projected/9ac51789-e545-4331-ae29-f99e8150fc15-kube-api-access-v5pvx\") pod \"payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh\" (UID: \"9ac51789-e545-4331-ae29-f99e8150fc15\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh" Oct 30 20:58:12.959421 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.959358 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz\" (UID: \"9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz" Oct 30 20:58:12.959546 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:12.959461 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g286\" (UniqueName: \"kubernetes.io/projected/9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d-kube-api-access-5g286\") pod \"payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz\" (UID: \"9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz" Oct 30 20:58:13.077990 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:13.068453 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz\" (UID: \"9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz" Oct 30 20:58:13.077990 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:13.068568 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g286\" (UniqueName: \"kubernetes.io/projected/9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d-kube-api-access-5g286\") pod \"payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz\" (UID: \"9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz" Oct 30 20:58:13.077990 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:13.068750 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac51789-e545-4331-ae29-f99e8150fc15-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh\" (UID: \"9ac51789-e545-4331-ae29-f99e8150fc15\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh" Oct 30 20:58:13.077990 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:13.068809 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5pvx\" (UniqueName: \"kubernetes.io/projected/9ac51789-e545-4331-ae29-f99e8150fc15-kube-api-access-v5pvx\") pod \"payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh\" (UID: \"9ac51789-e545-4331-ae29-f99e8150fc15\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh" Oct 30 20:58:13.077990 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:13.074303 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz\" (UID: \"9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz" Oct 30 20:58:13.077990 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:13.074648 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac51789-e545-4331-ae29-f99e8150fc15-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh\" (UID: \"9ac51789-e545-4331-ae29-f99e8150fc15\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh" Oct 30 20:58:13.164899 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:13.151520 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c4e64ec-eb49-4a9e-86df-7483d597fd9c" path="/var/lib/kubelet/pods/0c4e64ec-eb49-4a9e-86df-7483d597fd9c/volumes" Oct 30 20:58:13.164899 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:13.152108 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz"] Oct 30 20:58:13.392228 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:13.392185 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g286\" (UniqueName: \"kubernetes.io/projected/9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d-kube-api-access-5g286\") pod \"payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz\" (UID: \"9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz" Oct 30 20:58:13.442819 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:13.442782 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5pvx\" (UniqueName: \"kubernetes.io/projected/9ac51789-e545-4331-ae29-f99e8150fc15-kube-api-access-v5pvx\") pod \"payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh\" (UID: \"9ac51789-e545-4331-ae29-f99e8150fc15\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh" Oct 30 20:58:13.451392 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:13.446964 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz" Oct 30 20:58:13.473069 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:13.473040 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh" Oct 30 20:58:13.583002 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:13.582965 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn"] Oct 30 20:58:13.583997 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:13.583756 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" podUID="3f3e6ed8-9d1e-4716-ab89-a37141dc030d" containerName="istio-proxy" containerID="cri-o://c24557ce0d91c6c7907a6357bf8e52025fa5ff9da43cf1350d811d00167c5c17" gracePeriod=30 Oct 30 20:58:14.325113 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:14.324180 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh"] Oct 30 20:58:14.372201 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:14.366487 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b" Oct 30 20:58:14.372201 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:14.371456 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h" Oct 30 20:58:14.372540 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:14.372333 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz"] Oct 30 20:58:14.475725 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:14.475691 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz" event={"ID":"9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d","Type":"ContainerStarted","Data":"fcc60ba37e0929bfd69f381c89c16452c6a8b81678e617298efeab36b5a55561"} Oct 30 20:58:14.478110 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:14.477737 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh" event={"ID":"9ac51789-e545-4331-ae29-f99e8150fc15","Type":"ContainerStarted","Data":"78e7a0ad0a56ea8fe3c8faaa9e061ab51620ca9f3edede4de5dbaf754d276ca2"} Oct 30 20:58:14.478110 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:14.477831 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh" Oct 30 20:58:14.498165 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:14.498115 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh" podStartSLOduration=2.498101686 podStartE2EDuration="2.498101686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:14.492991006 +0000 UTC m=+1199.672887528" watchObservedRunningTime="2026-10-30 20:58:14.498101686 +0000 UTC m=+1199.677998193" Oct 30 20:58:15.279848 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.270207 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm"] Oct 30 20:58:15.373893 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.309182 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.373893 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.340193 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-955eb7-openshift-default-dockercfg-rrl6s\"" Oct 30 20:58:15.479614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.479576 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-credential-socket\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.480174 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.480148 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-istio-data\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.480372 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.480340 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-workload-socket\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.480488 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.480476 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-istio-envoy\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.480614 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.480601 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fef8496-2113-44ff-9694-038139e4b194-istio-token\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.480805 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.480737 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtb7h\" (UniqueName: \"kubernetes.io/projected/8fef8496-2113-44ff-9694-038139e4b194-kube-api-access-rtb7h\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.481014 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.480995 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fef8496-2113-44ff-9694-038139e4b194-istio-podinfo\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.481202 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.481172 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fef8496-2113-44ff-9694-038139e4b194-istiod-ca-cert\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.491331 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.491295 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-workload-certs\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.501948 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.501912 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm"] Oct 30 20:58:15.576566 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.576519 4848 generic.go:356] "Generic (PLEG): container finished" podID="16e788c7-69cc-48cb-959a-62c4763c2698" containerID="97484869dee74327eaf742fa17348a1abddd4ca77456e3d2017caa1046a662fd" exitCode=0 Oct 30 20:58:15.576758 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.576734 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" event={"ID":"16e788c7-69cc-48cb-959a-62c4763c2698","Type":"ContainerDied","Data":"97484869dee74327eaf742fa17348a1abddd4ca77456e3d2017caa1046a662fd"} Oct 30 20:58:15.593165 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.593125 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtb7h\" (UniqueName: \"kubernetes.io/projected/8fef8496-2113-44ff-9694-038139e4b194-kube-api-access-rtb7h\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.593338 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.593231 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fef8496-2113-44ff-9694-038139e4b194-istio-podinfo\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.593338 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.593288 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fef8496-2113-44ff-9694-038139e4b194-istiod-ca-cert\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.593451 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.593417 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-workload-certs\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.594388 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.593551 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-credential-socket\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.594388 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.593598 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-istio-data\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.594388 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.593717 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-workload-socket\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.594388 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.593759 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-istio-envoy\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.594703 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.593821 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fef8496-2113-44ff-9694-038139e4b194-istio-token\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.595807 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.595752 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-credential-socket\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.596092 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.596069 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-istio-data\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.596781 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.596746 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-workload-socket\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.597027 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.595503 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-workload-certs\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.598853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.598826 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fef8496-2113-44ff-9694-038139e4b194-istiod-ca-cert\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.599673 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.599647 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fef8496-2113-44ff-9694-038139e4b194-istio-podinfo\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.601024 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.600999 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-istio-envoy\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.645183 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.644381 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz" event={"ID":"9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d","Type":"ContainerStarted","Data":"c1cd4f243d2e2df8807d47c93c4b55f3c123bb7fe620cb135eee197f3f466745"} Oct 30 20:58:15.645183 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.644649 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz" Oct 30 20:58:15.649258 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.649157 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fef8496-2113-44ff-9694-038139e4b194-istio-token\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.650947 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.650744 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtb7h\" (UniqueName: \"kubernetes.io/projected/8fef8496-2113-44ff-9694-038139e4b194-kube-api-access-rtb7h\") pod \"e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.656898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.656869 4848 generic.go:356] "Generic (PLEG): container finished" podID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" containerID="1fee94e5820b62c606edde78e48dcedf152c59ecae177ace9c6929c5c40509a8" exitCode=0 Oct 30 20:58:15.657127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.657105 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" event={"ID":"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4","Type":"ContainerDied","Data":"1fee94e5820b62c606edde78e48dcedf152c59ecae177ace9c6929c5c40509a8"} Oct 30 20:58:15.666895 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.664910 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz" podStartSLOduration=3.66489356 podStartE2EDuration="3.66489356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:15.663962967 +0000 UTC m=+1200.843859474" watchObservedRunningTime="2026-10-30 20:58:15.66489356 +0000 UTC m=+1200.844790068" Oct 30 20:58:15.677194 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.677156 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh" event={"ID":"9ac51789-e545-4331-ae29-f99e8150fc15","Type":"ContainerStarted","Data":"1b5b30e87296d1d849d232dd0af5765e48ddedd6bb10cbb2a12f30862aaff0c8"} Oct 30 20:58:15.682924 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.682888 4848 generic.go:356] "Generic (PLEG): container finished" podID="883d31e2-ac6d-466c-900d-d86010b4028e" containerID="d96dd712b792441d9bd648e9df77a084aecdf2f904edcc3b3eadc340f3c64956" exitCode=0 Oct 30 20:58:15.683146 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.683013 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" event={"ID":"883d31e2-ac6d-466c-900d-d86010b4028e","Type":"ContainerDied","Data":"d96dd712b792441d9bd648e9df77a084aecdf2f904edcc3b3eadc340f3c64956"} Oct 30 20:58:15.929594 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.929186 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 20:58:15.934444 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.934418 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 20:58:15.944465 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.944430 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:15.945422 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.945389 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:58:15.947719 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.947697 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:58:15.955299 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.955277 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:58:15.962622 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.962598 4848 scope.go:122] "RemoveContainer" containerID="d96dd712b792441d9bd648e9df77a084aecdf2f904edcc3b3eadc340f3c64956" Oct 30 20:58:15.984472 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:15.983699 4848 scope.go:122] "RemoveContainer" containerID="97484869dee74327eaf742fa17348a1abddd4ca77456e3d2017caa1046a662fd" Oct 30 20:58:16.014789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.014085 4848 scope.go:122] "RemoveContainer" containerID="1fee94e5820b62c606edde78e48dcedf152c59ecae177ace9c6929c5c40509a8" Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.016073 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-data\") pod \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.016147 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-token\" (UniqueName: \"kubernetes.io/projected/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-token\") pod \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.016214 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-credential-socket\") pod \"883d31e2-ac6d-466c-900d-d86010b4028e\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.016244 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-envoy\") pod \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.016309 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-workload-socket\") pod \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.016364 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-podinfo\") pod \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.016400 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-istio-data\") pod \"16e788c7-69cc-48cb-959a-62c4763c2698\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.016454 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/883d31e2-ac6d-466c-900d-d86010b4028e-kube-api-access-btc44\" (UniqueName: \"kubernetes.io/projected/883d31e2-ac6d-466c-900d-d86010b4028e-kube-api-access-btc44\") pod \"883d31e2-ac6d-466c-900d-d86010b4028e\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.016520 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16e788c7-69cc-48cb-959a-62c4763c2698-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e788c7-69cc-48cb-959a-62c4763c2698-istio-podinfo\") pod \"16e788c7-69cc-48cb-959a-62c4763c2698\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.016550 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-workload-socket\") pod \"883d31e2-ac6d-466c-900d-d86010b4028e\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.016596 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-istio-data\") pod \"883d31e2-ac6d-466c-900d-d86010b4028e\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.016643 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-workload-socket\") pod \"16e788c7-69cc-48cb-959a-62c4763c2698\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.016689 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-credential-socket\") pod \"16e788c7-69cc-48cb-959a-62c4763c2698\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.016723 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e788c7-69cc-48cb-959a-62c4763c2698-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e788c7-69cc-48cb-959a-62c4763c2698-istiod-ca-cert\") pod \"16e788c7-69cc-48cb-959a-62c4763c2698\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.016790 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-credential-socket\") pod \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.016870 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-workload-certs\") pod \"16e788c7-69cc-48cb-959a-62c4763c2698\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.016906 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istiod-ca-cert\") pod \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.017026 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/883d31e2-ac6d-466c-900d-d86010b4028e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/883d31e2-ac6d-466c-900d-d86010b4028e-istiod-ca-cert\") pod \"883d31e2-ac6d-466c-900d-d86010b4028e\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.017071 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/883d31e2-ac6d-466c-900d-d86010b4028e-istio-token\" (UniqueName: \"kubernetes.io/projected/883d31e2-ac6d-466c-900d-d86010b4028e-istio-token\") pod \"883d31e2-ac6d-466c-900d-d86010b4028e\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.017122 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-workload-certs\") pod \"883d31e2-ac6d-466c-900d-d86010b4028e\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.017151 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-istio-envoy\") pod \"883d31e2-ac6d-466c-900d-d86010b4028e\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.017264 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e788c7-69cc-48cb-959a-62c4763c2698-istio-token\" (UniqueName: \"kubernetes.io/projected/16e788c7-69cc-48cb-959a-62c4763c2698-istio-token\") pod \"16e788c7-69cc-48cb-959a-62c4763c2698\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.017301 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e788c7-69cc-48cb-959a-62c4763c2698-kube-api-access-rm4xv\" (UniqueName: \"kubernetes.io/projected/16e788c7-69cc-48cb-959a-62c4763c2698-kube-api-access-rm4xv\") pod \"16e788c7-69cc-48cb-959a-62c4763c2698\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.017385 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-workload-certs\") pod \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.017448 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-kube-api-access-z7w4t\" (UniqueName: \"kubernetes.io/projected/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-kube-api-access-z7w4t\") pod \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\" (UID: \"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4\") " Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.018837 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-data" pod "0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" (UID: "0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.019056 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-workload-certs" pod "16e788c7-69cc-48cb-959a-62c4763c2698" (UID: "16e788c7-69cc-48cb-959a-62c4763c2698"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.019355 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istiod-ca-cert" pod "0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" (UID: "0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:16.020813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.019700 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/883d31e2-ac6d-466c-900d-d86010b4028e-istiod-ca-cert" pod "883d31e2-ac6d-466c-900d-d86010b4028e" (UID: "883d31e2-ac6d-466c-900d-d86010b4028e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:16.050833 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.032933 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-credential-socket" pod "0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" (UID: "0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:16.050833 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.035477 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-workload-certs" pod "0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" (UID: "0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:16.050833 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.035689 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-workload-certs" pod "883d31e2-ac6d-466c-900d-d86010b4028e" (UID: "883d31e2-ac6d-466c-900d-d86010b4028e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:16.050833 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.041963 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-workload-socket" pod "0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" (UID: "0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:16.052054 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.051375 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-workload-socket" pod "16e788c7-69cc-48cb-959a-62c4763c2698" (UID: "16e788c7-69cc-48cb-959a-62c4763c2698"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:16.052054 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.051467 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-istio-data" pod "16e788c7-69cc-48cb-959a-62c4763c2698" (UID: "16e788c7-69cc-48cb-959a-62c4763c2698"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:16.052054 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.051615 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-workload-socket" pod "883d31e2-ac6d-466c-900d-d86010b4028e" (UID: "883d31e2-ac6d-466c-900d-d86010b4028e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:16.052054 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.051658 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-istio-data" pod "883d31e2-ac6d-466c-900d-d86010b4028e" (UID: "883d31e2-ac6d-466c-900d-d86010b4028e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:16.052054 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.051933 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-credential-socket" pod "16e788c7-69cc-48cb-959a-62c4763c2698" (UID: "16e788c7-69cc-48cb-959a-62c4763c2698"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:16.052054 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.052011 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e788c7-69cc-48cb-959a-62c4763c2698-istiod-ca-cert" pod "16e788c7-69cc-48cb-959a-62c4763c2698" (UID: "16e788c7-69cc-48cb-959a-62c4763c2698"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:16.053911 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.053883 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-credential-socket" pod "883d31e2-ac6d-466c-900d-d86010b4028e" (UID: "883d31e2-ac6d-466c-900d-d86010b4028e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:16.056856 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.056812 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-podinfo" pod "0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" (UID: "0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:16.056856 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.056838 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883d31e2-ac6d-466c-900d-d86010b4028e-istio-token" pod "883d31e2-ac6d-466c-900d-d86010b4028e" (UID: "883d31e2-ac6d-466c-900d-d86010b4028e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:16.057157 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.057002 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-istio-envoy" pod "883d31e2-ac6d-466c-900d-d86010b4028e" (UID: "883d31e2-ac6d-466c-900d-d86010b4028e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:16.057157 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.057121 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-token" pod "0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" (UID: "0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:16.058247 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.058211 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16e788c7-69cc-48cb-959a-62c4763c2698-istio-podinfo" pod "16e788c7-69cc-48cb-959a-62c4763c2698" (UID: "16e788c7-69cc-48cb-959a-62c4763c2698"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:16.059197 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.059171 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e788c7-69cc-48cb-959a-62c4763c2698-istio-token" pod "16e788c7-69cc-48cb-959a-62c4763c2698" (UID: "16e788c7-69cc-48cb-959a-62c4763c2698"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:16.060551 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.060520 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e788c7-69cc-48cb-959a-62c4763c2698-kube-api-access-rm4xv" pod "16e788c7-69cc-48cb-959a-62c4763c2698" (UID: "16e788c7-69cc-48cb-959a-62c4763c2698"). InnerVolumeSpecName "kube-api-access-rm4xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:16.062044 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.061994 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-envoy" pod "0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" (UID: "0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:16.062044 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.062019 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-kube-api-access-z7w4t" pod "0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" (UID: "0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4"). InnerVolumeSpecName "kube-api-access-z7w4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:16.073274 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.073249 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883d31e2-ac6d-466c-900d-d86010b4028e-kube-api-access-btc44" pod "883d31e2-ac6d-466c-900d-d86010b4028e" (UID: "883d31e2-ac6d-466c-900d-d86010b4028e"). InnerVolumeSpecName "kube-api-access-btc44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.144529 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/883d31e2-ac6d-466c-900d-d86010b4028e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/883d31e2-ac6d-466c-900d-d86010b4028e-istio-podinfo\") pod \"883d31e2-ac6d-466c-900d-d86010b4028e\" (UID: \"883d31e2-ac6d-466c-900d-d86010b4028e\") " Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.144603 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-istio-envoy\") pod \"16e788c7-69cc-48cb-959a-62c4763c2698\" (UID: \"16e788c7-69cc-48cb-959a-62c4763c2698\") " Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145374 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16e788c7-69cc-48cb-959a-62c4763c2698-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145401 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145416 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145429 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145442 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145455 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16e788c7-69cc-48cb-959a-62c4763c2698-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145468 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145481 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145494 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145507 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/883d31e2-ac6d-466c-900d-d86010b4028e-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145522 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/883d31e2-ac6d-466c-900d-d86010b4028e-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145536 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145559 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145572 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16e788c7-69cc-48cb-959a-62c4763c2698-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145585 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm4xv\" (UniqueName: \"kubernetes.io/projected/16e788c7-69cc-48cb-959a-62c4763c2698-kube-api-access-rm4xv\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145599 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145612 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7w4t\" (UniqueName: \"kubernetes.io/projected/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-kube-api-access-z7w4t\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145627 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145640 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145653 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/883d31e2-ac6d-466c-900d-d86010b4028e-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145665 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145680 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145693 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145705 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.150898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.145717 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btc44\" (UniqueName: \"kubernetes.io/projected/883d31e2-ac6d-466c-900d-d86010b4028e-kube-api-access-btc44\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.157451 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.157425 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-istio-envoy" pod "16e788c7-69cc-48cb-959a-62c4763c2698" (UID: "16e788c7-69cc-48cb-959a-62c4763c2698"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:16.161976 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.161933 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/883d31e2-ac6d-466c-900d-d86010b4028e-istio-podinfo" pod "883d31e2-ac6d-466c-900d-d86010b4028e" (UID: "883d31e2-ac6d-466c-900d-d86010b4028e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:16.247791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.247744 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/883d31e2-ac6d-466c-900d-d86010b4028e-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.247791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.247786 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16e788c7-69cc-48cb-959a-62c4763c2698-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:16.297424 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.297394 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm"] Oct 30 20:58:16.298309 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:58:16.298288 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fef8496_2113_44ff_9694_038139e4b194.slice/crio-a91b9a84d66beda3f5ce7e3f907d3ec76d9676577849312b4550912e74958cec WatchSource:0}: Error finding container a91b9a84d66beda3f5ce7e3f907d3ec76d9676577849312b4550912e74958cec: Status 404 returned error can't find the container with id a91b9a84d66beda3f5ce7e3f907d3ec76d9676577849312b4550912e74958cec Oct 30 20:58:16.300921 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.300889 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:16.300978 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.300958 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:16.301010 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.300989 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:16.690191 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.690152 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" event={"ID":"0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4","Type":"ContainerDied","Data":"910a1e31359dcb8e752608741f450f9916c73dd1692947b5b7f0123b6454bab1"} Oct 30 20:58:16.690627 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.690223 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" event={"ID":"883d31e2-ac6d-466c-900d-d86010b4028e","Type":"ContainerDied","Data":"75ceebc0f3a4e5dc3ffc72262346231ea76536a4db6e808b1246198708e6b1c8"} Oct 30 20:58:16.692257 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.692229 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" event={"ID":"8fef8496-2113-44ff-9694-038139e4b194","Type":"ContainerStarted","Data":"fd6a60993c23f9881c200d1afe9b1f5c92930b5fb5b92a8cd853a88518ef0d85"} Oct 30 20:58:16.692405 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.692285 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" event={"ID":"8fef8496-2113-44ff-9694-038139e4b194","Type":"ContainerStarted","Data":"a91b9a84d66beda3f5ce7e3f907d3ec76d9676577849312b4550912e74958cec"} Oct 30 20:58:16.692405 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.692312 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" event={"ID":"16e788c7-69cc-48cb-959a-62c4763c2698","Type":"ContainerDied","Data":"fad9eb3a9a6e5c655857f7f34c42af038f0595d06a1eef8b24b2cf7cde2dd472"} Oct 30 20:58:16.692500 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.692425 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs" Oct 30 20:58:16.692500 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.692445 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m" Oct 30 20:58:16.692577 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.692557 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx" Oct 30 20:58:16.752788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.752646 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" podStartSLOduration=1.752631961 podStartE2EDuration="1.752631961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:16.719059934 +0000 UTC m=+1201.898956442" watchObservedRunningTime="2026-10-30 20:58:16.752631961 +0000 UTC m=+1201.932528463" Oct 30 20:58:16.761269 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.761239 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs"] Oct 30 20:58:16.766757 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.766731 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-w6qrs"] Oct 30 20:58:16.793674 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.793644 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m"] Oct 30 20:58:16.799400 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.799370 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-t686m"] Oct 30 20:58:16.856279 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.856242 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx"] Oct 30 20:58:16.862458 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.862421 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-gpqhx"] Oct 30 20:58:16.997696 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.997658 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" path="/var/lib/kubelet/pods/0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4/volumes" Oct 30 20:58:16.998584 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.998561 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e788c7-69cc-48cb-959a-62c4763c2698" path="/var/lib/kubelet/pods/16e788c7-69cc-48cb-959a-62c4763c2698/volumes" Oct 30 20:58:16.999387 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:16.999368 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="883d31e2-ac6d-466c-900d-d86010b4028e" path="/var/lib/kubelet/pods/883d31e2-ac6d-466c-900d-d86010b4028e/volumes" Oct 30 20:58:17.013222 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.013144 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:17.650897 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.649273 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd"] Oct 30 20:58:17.650897 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.649525 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" podUID="932dfb46-276c-4d55-9800-f1cbaf96360d" containerName="istio-proxy" containerID="cri-o://cd3adba4e5182237fcf71de4636acd6c3e4f9f919ffef0e56881a5df452db327" gracePeriod=30 Oct 30 20:58:17.663820 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.660274 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j"] Oct 30 20:58:17.663820 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.660535 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" podUID="a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" containerName="istio-proxy" containerID="cri-o://6272471aa9cda5bc96709e1a2be742aa7e1b0a63ff6cf268b7c7add707e7f06f" gracePeriod=30 Oct 30 20:58:17.669780 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.669737 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz"] Oct 30 20:58:17.671557 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.670131 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" podUID="3a9a5360-6f89-4b28-9638-a56fdb01389f" containerName="istio-proxy" containerID="cri-o://8e520978851e1ebba7ed8eff2dc733057244d6c65c632e1b70d3a5db6b8c7036" gracePeriod=30 Oct 30 20:58:17.682261 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.682233 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:58:17.760605 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.759544 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-envoy\") pod \"96a95230-aa96-4d6b-856e-e29871f3d3bf\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " Oct 30 20:58:17.761530 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.761506 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-credential-socket\") pod \"96a95230-aa96-4d6b-856e-e29871f3d3bf\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " Oct 30 20:58:17.761834 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.761753 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a95230-aa96-4d6b-856e-e29871f3d3bf-kube-api-access-vvwgb\" (UniqueName: \"kubernetes.io/projected/96a95230-aa96-4d6b-856e-e29871f3d3bf-kube-api-access-vvwgb\") pod \"96a95230-aa96-4d6b-856e-e29871f3d3bf\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " Oct 30 20:58:17.762105 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.762085 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-token\" (UniqueName: \"kubernetes.io/projected/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-token\") pod \"96a95230-aa96-4d6b-856e-e29871f3d3bf\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " Oct 30 20:58:17.762392 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.762374 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-workload-certs\") pod \"96a95230-aa96-4d6b-856e-e29871f3d3bf\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " Oct 30 20:58:17.762654 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.762639 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-workload-socket\") pod \"96a95230-aa96-4d6b-856e-e29871f3d3bf\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " Oct 30 20:58:17.762890 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.761869 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-credential-socket" pod "96a95230-aa96-4d6b-856e-e29871f3d3bf" (UID: "96a95230-aa96-4d6b-856e-e29871f3d3bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:17.763049 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.763033 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-data\") pod \"96a95230-aa96-4d6b-856e-e29871f3d3bf\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " Oct 30 20:58:17.763197 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.763183 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a95230-aa96-4d6b-856e-e29871f3d3bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a95230-aa96-4d6b-856e-e29871f3d3bf-istiod-ca-cert\") pod \"96a95230-aa96-4d6b-856e-e29871f3d3bf\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " Oct 30 20:58:17.763381 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.763111 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-workload-certs" pod "96a95230-aa96-4d6b-856e-e29871f3d3bf" (UID: "96a95230-aa96-4d6b-856e-e29871f3d3bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:17.763381 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.763337 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-podinfo\") pod \"96a95230-aa96-4d6b-856e-e29871f3d3bf\" (UID: \"96a95230-aa96-4d6b-856e-e29871f3d3bf\") " Oct 30 20:58:17.763381 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.763235 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-workload-socket" pod "96a95230-aa96-4d6b-856e-e29871f3d3bf" (UID: "96a95230-aa96-4d6b-856e-e29871f3d3bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:17.763381 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.763317 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-data" pod "96a95230-aa96-4d6b-856e-e29871f3d3bf" (UID: "96a95230-aa96-4d6b-856e-e29871f3d3bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:17.763709 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.763583 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a95230-aa96-4d6b-856e-e29871f3d3bf-istiod-ca-cert" pod "96a95230-aa96-4d6b-856e-e29871f3d3bf" (UID: "96a95230-aa96-4d6b-856e-e29871f3d3bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:17.767473 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.764182 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:17.767473 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.764209 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96a95230-aa96-4d6b-856e-e29871f3d3bf-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:17.767473 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.764224 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:17.767473 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.764239 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:17.767473 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.764254 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:17.769761 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.769734 4848 generic.go:356] "Generic (PLEG): container finished" podID="96a95230-aa96-4d6b-856e-e29871f3d3bf" containerID="ac1fdb670c5c8dfb173984560945b31220d115e913cc25d7cbcc23a47b413f15" exitCode=0 Oct 30 20:58:17.769910 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.769822 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" Oct 30 20:58:17.769910 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.769898 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" event={"ID":"96a95230-aa96-4d6b-856e-e29871f3d3bf","Type":"ContainerDied","Data":"ac1fdb670c5c8dfb173984560945b31220d115e913cc25d7cbcc23a47b413f15"} Oct 30 20:58:17.770021 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.769948 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5" event={"ID":"96a95230-aa96-4d6b-856e-e29871f3d3bf","Type":"ContainerDied","Data":"c2ac109843844336377c229a6c5fec5e76b11ab85648281354b8a4d35530fc31"} Oct 30 20:58:17.770021 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.769976 4848 scope.go:122] "RemoveContainer" containerID="ac1fdb670c5c8dfb173984560945b31220d115e913cc25d7cbcc23a47b413f15" Oct 30 20:58:17.772815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.772633 4848 generic.go:356] "Generic (PLEG): container finished" podID="794057dc-d92a-4c15-8003-d3e8132297da" containerID="1f64d311a40d18931f63c5037f1375d1ea0e552d26dc166786963a74d0e63c74" exitCode=0 Oct 30 20:58:17.772815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.772728 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" event={"ID":"794057dc-d92a-4c15-8003-d3e8132297da","Type":"ContainerDied","Data":"1f64d311a40d18931f63c5037f1375d1ea0e552d26dc166786963a74d0e63c74"} Oct 30 20:58:17.776097 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.775092 4848 generic.go:356] "Generic (PLEG): container finished" podID="f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" containerID="a20a378097eceae6fc8987bbc2b885b752fb17ef981f1359169809adfbcf8bbc" exitCode=0 Oct 30 20:58:17.776097 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.775972 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" event={"ID":"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d","Type":"ContainerDied","Data":"a20a378097eceae6fc8987bbc2b885b752fb17ef981f1359169809adfbcf8bbc"} Oct 30 20:58:17.802539 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.796396 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-token" pod "96a95230-aa96-4d6b-856e-e29871f3d3bf" (UID: "96a95230-aa96-4d6b-856e-e29871f3d3bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:17.802539 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.796554 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-envoy" pod "96a95230-aa96-4d6b-856e-e29871f3d3bf" (UID: "96a95230-aa96-4d6b-856e-e29871f3d3bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:17.802539 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.797303 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a95230-aa96-4d6b-856e-e29871f3d3bf-kube-api-access-vvwgb" pod "96a95230-aa96-4d6b-856e-e29871f3d3bf" (UID: "96a95230-aa96-4d6b-856e-e29871f3d3bf"). InnerVolumeSpecName "kube-api-access-vvwgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:17.802539 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.799958 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-podinfo" pod "96a95230-aa96-4d6b-856e-e29871f3d3bf" (UID: "96a95230-aa96-4d6b-856e-e29871f3d3bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:17.845848 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.845162 4848 scope.go:122] "RemoveContainer" containerID="ac1fdb670c5c8dfb173984560945b31220d115e913cc25d7cbcc23a47b413f15" Oct 30 20:58:17.845848 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:58:17.845552 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1fdb670c5c8dfb173984560945b31220d115e913cc25d7cbcc23a47b413f15\": container with ID starting with ac1fdb670c5c8dfb173984560945b31220d115e913cc25d7cbcc23a47b413f15 not found: ID does not exist" containerID="ac1fdb670c5c8dfb173984560945b31220d115e913cc25d7cbcc23a47b413f15" Oct 30 20:58:17.845848 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.845586 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1fdb670c5c8dfb173984560945b31220d115e913cc25d7cbcc23a47b413f15"} err="failed to get container status \"ac1fdb670c5c8dfb173984560945b31220d115e913cc25d7cbcc23a47b413f15\": rpc error: code = NotFound desc = could not find container \"ac1fdb670c5c8dfb173984560945b31220d115e913cc25d7cbcc23a47b413f15\": container with ID starting with ac1fdb670c5c8dfb173984560945b31220d115e913cc25d7cbcc23a47b413f15 not found: ID does not exist" Oct 30 20:58:17.866478 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.866444 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:17.866616 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.866482 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:17.866616 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.866497 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvwgb\" (UniqueName: \"kubernetes.io/projected/96a95230-aa96-4d6b-856e-e29871f3d3bf-kube-api-access-vvwgb\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:17.866616 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.866512 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96a95230-aa96-4d6b-856e-e29871f3d3bf-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:17.903896 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.903629 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:58:17.952121 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.952086 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:17.959242 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:17.959209 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:58:18.069086 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069051 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-istio-envoy\") pod \"794057dc-d92a-4c15-8003-d3e8132297da\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " Oct 30 20:58:18.069237 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069108 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-credential-socket\") pod \"794057dc-d92a-4c15-8003-d3e8132297da\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " Oct 30 20:58:18.069237 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069179 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/794057dc-d92a-4c15-8003-d3e8132297da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794057dc-d92a-4c15-8003-d3e8132297da-istiod-ca-cert\") pod \"794057dc-d92a-4c15-8003-d3e8132297da\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " Oct 30 20:58:18.069237 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069211 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-credential-socket\") pod \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " Oct 30 20:58:18.069374 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069265 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-token\" (UniqueName: \"kubernetes.io/projected/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-token\") pod \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " Oct 30 20:58:18.069374 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069302 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-podinfo\") pod \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " Oct 30 20:58:18.069374 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069339 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-workload-certs\") pod \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " Oct 30 20:58:18.069547 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069382 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/794057dc-d92a-4c15-8003-d3e8132297da-istio-token\" (UniqueName: \"kubernetes.io/projected/794057dc-d92a-4c15-8003-d3e8132297da-istio-token\") pod \"794057dc-d92a-4c15-8003-d3e8132297da\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " Oct 30 20:58:18.069547 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069427 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-data\") pod \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " Oct 30 20:58:18.069547 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069475 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-credential-socket" pod "794057dc-d92a-4c15-8003-d3e8132297da" (UID: "794057dc-d92a-4c15-8003-d3e8132297da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:18.069547 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069517 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-credential-socket" pod "f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" (UID: "f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:18.069547 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069529 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-envoy\") pod \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " Oct 30 20:58:18.069823 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069588 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-istio-data\") pod \"794057dc-d92a-4c15-8003-d3e8132297da\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " Oct 30 20:58:18.069823 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069645 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/794057dc-d92a-4c15-8003-d3e8132297da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794057dc-d92a-4c15-8003-d3e8132297da-istio-podinfo\") pod \"794057dc-d92a-4c15-8003-d3e8132297da\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " Oct 30 20:58:18.069823 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069683 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-workload-certs" pod "f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" (UID: "f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:18.069823 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069692 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-workload-socket\") pod \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " Oct 30 20:58:18.069823 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069699 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/794057dc-d92a-4c15-8003-d3e8132297da-istiod-ca-cert" pod "794057dc-d92a-4c15-8003-d3e8132297da" (UID: "794057dc-d92a-4c15-8003-d3e8132297da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:18.069823 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069755 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/794057dc-d92a-4c15-8003-d3e8132297da-kube-api-access-5t2d9\" (UniqueName: \"kubernetes.io/projected/794057dc-d92a-4c15-8003-d3e8132297da-kube-api-access-5t2d9\") pod \"794057dc-d92a-4c15-8003-d3e8132297da\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " Oct 30 20:58:18.070123 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069831 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-workload-socket\") pod \"794057dc-d92a-4c15-8003-d3e8132297da\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " Oct 30 20:58:18.070123 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069908 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istiod-ca-cert\") pod \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " Oct 30 20:58:18.070123 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069921 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-workload-socket" pod "f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" (UID: "f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:18.070123 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069969 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-data" pod "f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" (UID: "f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:18.070123 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.069992 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-kube-api-access-b9ndm\" (UniqueName: \"kubernetes.io/projected/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-kube-api-access-b9ndm\") pod \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\" (UID: \"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d\") " Oct 30 20:58:18.070123 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.070102 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-istio-data" pod "794057dc-d92a-4c15-8003-d3e8132297da" (UID: "794057dc-d92a-4c15-8003-d3e8132297da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:18.070123 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.070109 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-workload-certs\") pod \"794057dc-d92a-4c15-8003-d3e8132297da\" (UID: \"794057dc-d92a-4c15-8003-d3e8132297da\") " Oct 30 20:58:18.070441 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.070225 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-workload-socket" pod "794057dc-d92a-4c15-8003-d3e8132297da" (UID: "794057dc-d92a-4c15-8003-d3e8132297da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:18.070797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.070756 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:18.070928 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.070852 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:18.070928 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.070867 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:18.070928 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.070881 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:18.070928 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.070895 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:18.070928 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.070910 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:18.070928 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.070924 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/794057dc-d92a-4c15-8003-d3e8132297da-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:18.071322 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.070938 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:18.071649 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.071623 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-workload-certs" pod "794057dc-d92a-4c15-8003-d3e8132297da" (UID: "794057dc-d92a-4c15-8003-d3e8132297da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:18.071874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.071846 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-envoy" pod "f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" (UID: "f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:18.072017 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.071994 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istiod-ca-cert" pod "f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" (UID: "f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:18.072087 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.072070 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-istio-envoy" pod "794057dc-d92a-4c15-8003-d3e8132297da" (UID: "794057dc-d92a-4c15-8003-d3e8132297da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:18.072188 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.072167 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/794057dc-d92a-4c15-8003-d3e8132297da-istio-podinfo" pod "794057dc-d92a-4c15-8003-d3e8132297da" (UID: "794057dc-d92a-4c15-8003-d3e8132297da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:18.072506 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.072483 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794057dc-d92a-4c15-8003-d3e8132297da-kube-api-access-5t2d9" pod "794057dc-d92a-4c15-8003-d3e8132297da" (UID: "794057dc-d92a-4c15-8003-d3e8132297da"). InnerVolumeSpecName "kube-api-access-5t2d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:18.072820 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.072797 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794057dc-d92a-4c15-8003-d3e8132297da-istio-token" pod "794057dc-d92a-4c15-8003-d3e8132297da" (UID: "794057dc-d92a-4c15-8003-d3e8132297da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:18.073759 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.073725 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-kube-api-access-b9ndm" pod "f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" (UID: "f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d"). InnerVolumeSpecName "kube-api-access-b9ndm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:18.074403 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.074380 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-token" pod "f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" (UID: "f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:18.074496 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.074446 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-podinfo" pod "f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" (UID: "f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:18.099024 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.098991 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5"] Oct 30 20:58:18.103079 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.103054 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-42rg5"] Oct 30 20:58:18.171857 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.171822 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:18.171857 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.171851 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:18.171857 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.171864 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794057dc-d92a-4c15-8003-d3e8132297da-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:18.172152 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.171873 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:18.172152 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.171885 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/794057dc-d92a-4c15-8003-d3e8132297da-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:18.172152 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.171898 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t2d9\" (UniqueName: \"kubernetes.io/projected/794057dc-d92a-4c15-8003-d3e8132297da-kube-api-access-5t2d9\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:18.172152 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.171908 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:18.172152 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.171918 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9ndm\" (UniqueName: \"kubernetes.io/projected/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d-kube-api-access-b9ndm\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:18.172152 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.171926 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:18.172152 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.171937 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/794057dc-d92a-4c15-8003-d3e8132297da-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:18.930748 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.918158 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" event={"ID":"794057dc-d92a-4c15-8003-d3e8132297da","Type":"ContainerDied","Data":"ecebe43270f24fa3e2a333f2e30b4ffc916a471b703424a9c687bc9e834eca93"} Oct 30 20:58:18.930748 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.918259 4848 scope.go:122] "RemoveContainer" containerID="1f64d311a40d18931f63c5037f1375d1ea0e552d26dc166786963a74d0e63c74" Oct 30 20:58:18.930748 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.918406 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx" Oct 30 20:58:18.930748 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.923611 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" event={"ID":"f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d","Type":"ContainerDied","Data":"2974bf64e1fd92e77ece5c2099ec121696c836100aed7ddba067bd821ae25506"} Oct 30 20:58:18.930748 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.923736 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb" Oct 30 20:58:18.948179 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.948157 4848 scope.go:122] "RemoveContainer" containerID="a20a378097eceae6fc8987bbc2b885b752fb17ef981f1359169809adfbcf8bbc" Oct 30 20:58:18.952684 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.952655 4848 generic.go:356] "Generic (PLEG): container finished" podID="3f3e6ed8-9d1e-4716-ab89-a37141dc030d" containerID="c24557ce0d91c6c7907a6357bf8e52025fa5ff9da43cf1350d811d00167c5c17" exitCode=0 Oct 30 20:58:18.955402 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.955376 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a95230-aa96-4d6b-856e-e29871f3d3bf" path="/var/lib/kubelet/pods/96a95230-aa96-4d6b-856e-e29871f3d3bf/volumes" Oct 30 20:58:18.956333 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.956315 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:18.956500 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.956481 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" event={"ID":"3f3e6ed8-9d1e-4716-ab89-a37141dc030d","Type":"ContainerDied","Data":"c24557ce0d91c6c7907a6357bf8e52025fa5ff9da43cf1350d811d00167c5c17"} Oct 30 20:58:18.957235 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.957209 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:58:18.973915 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.973890 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx"] Oct 30 20:58:18.975249 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.975227 4848 scope.go:122] "RemoveContainer" containerID="29be838c4e202a929f01d3ae03df0128512314e8de2d5fd95bea31235a40a046" Oct 30 20:58:18.980506 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:18.980462 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-n9gcx"] Oct 30 20:58:19.017830 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.017573 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb"] Oct 30 20:58:19.023343 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.023312 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-jmjqb"] Oct 30 20:58:19.041723 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.041694 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:58:19.093865 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.093828 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-envoy\") pod \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " Oct 30 20:58:19.094045 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.093874 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-data\") pod \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " Oct 30 20:58:19.094045 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.093934 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-podinfo\") pod \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " Oct 30 20:58:19.094045 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.094016 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-workload-socket\") pod \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " Oct 30 20:58:19.094218 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.094111 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-credential-socket\") pod \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " Oct 30 20:58:19.094218 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.094168 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istiod-ca-cert\") pod \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " Oct 30 20:58:19.094218 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.094202 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-token\" (UniqueName: \"kubernetes.io/projected/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-token\") pod \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " Oct 30 20:58:19.094218 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.094199 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-data" pod "3f3e6ed8-9d1e-4716-ab89-a37141dc030d" (UID: "3f3e6ed8-9d1e-4716-ab89-a37141dc030d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:19.094468 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.094345 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-credential-socket" pod "3f3e6ed8-9d1e-4716-ab89-a37141dc030d" (UID: "3f3e6ed8-9d1e-4716-ab89-a37141dc030d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:19.094468 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.094362 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-workload-certs\") pod \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " Oct 30 20:58:19.094577 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.094466 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-kube-api-access-t8bgx\" (UniqueName: \"kubernetes.io/projected/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-kube-api-access-t8bgx\") pod \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\" (UID: \"3f3e6ed8-9d1e-4716-ab89-a37141dc030d\") " Oct 30 20:58:19.094577 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.094516 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istiod-ca-cert" pod "3f3e6ed8-9d1e-4716-ab89-a37141dc030d" (UID: "3f3e6ed8-9d1e-4716-ab89-a37141dc030d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:19.094838 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.094788 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-workload-certs" pod "3f3e6ed8-9d1e-4716-ab89-a37141dc030d" (UID: "3f3e6ed8-9d1e-4716-ab89-a37141dc030d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:19.095044 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.095019 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:19.095044 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.095045 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:19.095216 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.095060 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:19.095216 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.095075 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:19.095363 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.095341 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-workload-socket" pod "3f3e6ed8-9d1e-4716-ab89-a37141dc030d" (UID: "3f3e6ed8-9d1e-4716-ab89-a37141dc030d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:19.096390 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.096369 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-podinfo" pod "3f3e6ed8-9d1e-4716-ab89-a37141dc030d" (UID: "3f3e6ed8-9d1e-4716-ab89-a37141dc030d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:19.096918 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.096900 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-envoy" pod "3f3e6ed8-9d1e-4716-ab89-a37141dc030d" (UID: "3f3e6ed8-9d1e-4716-ab89-a37141dc030d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:19.097148 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.097129 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-token" pod "3f3e6ed8-9d1e-4716-ab89-a37141dc030d" (UID: "3f3e6ed8-9d1e-4716-ab89-a37141dc030d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:19.097199 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.097157 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-kube-api-access-t8bgx" pod "3f3e6ed8-9d1e-4716-ab89-a37141dc030d" (UID: "3f3e6ed8-9d1e-4716-ab89-a37141dc030d"). InnerVolumeSpecName "kube-api-access-t8bgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:19.196509 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.196421 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8bgx\" (UniqueName: \"kubernetes.io/projected/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-kube-api-access-t8bgx\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:19.196509 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.196450 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:19.196509 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.196462 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:19.196509 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.196471 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:19.196509 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.196479 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f3e6ed8-9d1e-4716-ab89-a37141dc030d-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:19.967406 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.967370 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" event={"ID":"3f3e6ed8-9d1e-4716-ab89-a37141dc030d","Type":"ContainerDied","Data":"d497f3c6c09f4db3061fe658200d5f3b3e1a5239ac337e6c16bfaa435cfe4794"} Oct 30 20:58:19.967910 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.967440 4848 scope.go:122] "RemoveContainer" containerID="c24557ce0d91c6c7907a6357bf8e52025fa5ff9da43cf1350d811d00167c5c17" Oct 30 20:58:19.967910 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:19.967383 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn" Oct 30 20:58:20.038710 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.038674 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn"] Oct 30 20:58:20.059960 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.059924 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5452c3-openshift-default-7bcf44b967-z8chn"] Oct 30 20:58:20.522262 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.522230 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx"] Oct 30 20:58:20.522871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.522847 4848 cpu_manager.go:414] "containerMap: removing container" podUID="883d31e2-ac6d-466c-900d-d86010b4028e" containerName="istio-proxy" Oct 30 20:58:20.522871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.522866 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="883d31e2-ac6d-466c-900d-d86010b4028e" containerName="istio-proxy" Oct 30 20:58:20.523066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.522881 4848 cpu_manager.go:414] "containerMap: removing container" podUID="16e788c7-69cc-48cb-959a-62c4763c2698" containerName="istio-proxy" Oct 30 20:58:20.523066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.522886 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e788c7-69cc-48cb-959a-62c4763c2698" containerName="istio-proxy" Oct 30 20:58:20.523066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.522898 4848 cpu_manager.go:414] "containerMap: removing container" podUID="3f3e6ed8-9d1e-4716-ab89-a37141dc030d" containerName="istio-proxy" Oct 30 20:58:20.523066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.522903 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f3e6ed8-9d1e-4716-ab89-a37141dc030d" containerName="istio-proxy" Oct 30 20:58:20.523066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.522915 4848 cpu_manager.go:414] "containerMap: removing container" podUID="f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" containerName="istio-proxy" Oct 30 20:58:20.523066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.522920 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" containerName="istio-proxy" Oct 30 20:58:20.523066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.522936 4848 cpu_manager.go:414] "containerMap: removing container" podUID="794057dc-d92a-4c15-8003-d3e8132297da" containerName="istio-proxy" Oct 30 20:58:20.523066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.522941 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794057dc-d92a-4c15-8003-d3e8132297da" containerName="istio-proxy" Oct 30 20:58:20.523066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.522950 4848 cpu_manager.go:414] "containerMap: removing container" podUID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" containerName="istio-proxy" Oct 30 20:58:20.523066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.522955 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" containerName="istio-proxy" Oct 30 20:58:20.523066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.522967 4848 cpu_manager.go:414] "containerMap: removing container" podUID="96a95230-aa96-4d6b-856e-e29871f3d3bf" containerName="istio-proxy" Oct 30 20:58:20.523066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.522971 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a95230-aa96-4d6b-856e-e29871f3d3bf" containerName="istio-proxy" Oct 30 20:58:20.523066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.523060 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="883d31e2-ac6d-466c-900d-d86010b4028e" containerName="istio-proxy" Oct 30 20:58:20.523066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.523071 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e788c7-69cc-48cb-959a-62c4763c2698" containerName="istio-proxy" Oct 30 20:58:20.523066 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.523078 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f3e6ed8-9d1e-4716-ab89-a37141dc030d" containerName="istio-proxy" Oct 30 20:58:20.524201 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.523085 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" containerName="istio-proxy" Oct 30 20:58:20.524201 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.523093 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="794057dc-d92a-4c15-8003-d3e8132297da" containerName="istio-proxy" Oct 30 20:58:20.524201 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.523099 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a91cf2e-5df1-4d3b-9557-d4b8f826d5a4" containerName="istio-proxy" Oct 30 20:58:20.524201 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.523106 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f3e6ed8-9d1e-4716-ab89-a37141dc030d" containerName="istio-proxy" Oct 30 20:58:20.524201 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.523112 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a95230-aa96-4d6b-856e-e29871f3d3bf" containerName="istio-proxy" Oct 30 20:58:20.526530 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.526510 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx" Oct 30 20:58:20.530716 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.530695 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-955eb7-dockercfg-2dh4n\"" Oct 30 20:58:20.533154 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.533135 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-955eb7\"" Oct 30 20:58:20.545732 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.545709 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx"] Oct 30 20:58:20.556958 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.556924 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd"] Oct 30 20:58:20.557888 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.557864 4848 cpu_manager.go:414] "containerMap: removing container" podUID="3f3e6ed8-9d1e-4716-ab89-a37141dc030d" containerName="istio-proxy" Oct 30 20:58:20.557888 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.557887 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f3e6ed8-9d1e-4716-ab89-a37141dc030d" containerName="istio-proxy" Oct 30 20:58:20.562035 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.562015 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd" Oct 30 20:58:20.574289 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.574267 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd"] Oct 30 20:58:20.627820 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.627756 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/598f3e3d-8117-4d80-b192-8fa7799a2420-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx\" (UID: \"598f3e3d-8117-4d80-b192-8fa7799a2420\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx" Oct 30 20:58:20.627969 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.627874 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22g25\" (UniqueName: \"kubernetes.io/projected/598f3e3d-8117-4d80-b192-8fa7799a2420-kube-api-access-22g25\") pod \"payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx\" (UID: \"598f3e3d-8117-4d80-b192-8fa7799a2420\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx" Oct 30 20:58:20.628032 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.627992 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk8v2\" (UniqueName: \"kubernetes.io/projected/5d07b7cf-5e5f-470d-82d8-6ff84370fdf3-kube-api-access-lk8v2\") pod \"payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd\" (UID: \"5d07b7cf-5e5f-470d-82d8-6ff84370fdf3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd" Oct 30 20:58:20.628032 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.628026 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d07b7cf-5e5f-470d-82d8-6ff84370fdf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd\" (UID: \"5d07b7cf-5e5f-470d-82d8-6ff84370fdf3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd" Oct 30 20:58:20.735910 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.732693 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk8v2\" (UniqueName: \"kubernetes.io/projected/5d07b7cf-5e5f-470d-82d8-6ff84370fdf3-kube-api-access-lk8v2\") pod \"payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd\" (UID: \"5d07b7cf-5e5f-470d-82d8-6ff84370fdf3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd" Oct 30 20:58:20.735910 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.732788 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d07b7cf-5e5f-470d-82d8-6ff84370fdf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd\" (UID: \"5d07b7cf-5e5f-470d-82d8-6ff84370fdf3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd" Oct 30 20:58:20.735910 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.732994 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/598f3e3d-8117-4d80-b192-8fa7799a2420-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx\" (UID: \"598f3e3d-8117-4d80-b192-8fa7799a2420\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx" Oct 30 20:58:20.735910 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.733025 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22g25\" (UniqueName: \"kubernetes.io/projected/598f3e3d-8117-4d80-b192-8fa7799a2420-kube-api-access-22g25\") pod \"payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx\" (UID: \"598f3e3d-8117-4d80-b192-8fa7799a2420\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx" Oct 30 20:58:20.735910 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.734368 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d07b7cf-5e5f-470d-82d8-6ff84370fdf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd\" (UID: \"5d07b7cf-5e5f-470d-82d8-6ff84370fdf3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd" Oct 30 20:58:20.735910 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.735116 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/598f3e3d-8117-4d80-b192-8fa7799a2420-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx\" (UID: \"598f3e3d-8117-4d80-b192-8fa7799a2420\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx" Oct 30 20:58:20.817460 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.815708 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22g25\" (UniqueName: \"kubernetes.io/projected/598f3e3d-8117-4d80-b192-8fa7799a2420-kube-api-access-22g25\") pod \"payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx\" (UID: \"598f3e3d-8117-4d80-b192-8fa7799a2420\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx" Oct 30 20:58:20.817460 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.816016 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk8v2\" (UniqueName: \"kubernetes.io/projected/5d07b7cf-5e5f-470d-82d8-6ff84370fdf3-kube-api-access-lk8v2\") pod \"payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd\" (UID: \"5d07b7cf-5e5f-470d-82d8-6ff84370fdf3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd" Oct 30 20:58:20.843943 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.843669 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx" Oct 30 20:58:20.879237 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.878983 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd" Oct 30 20:58:20.975859 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.975823 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f3e6ed8-9d1e-4716-ab89-a37141dc030d" path="/var/lib/kubelet/pods/3f3e6ed8-9d1e-4716-ab89-a37141dc030d/volumes" Oct 30 20:58:20.976617 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.976596 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="794057dc-d92a-4c15-8003-d3e8132297da" path="/var/lib/kubelet/pods/794057dc-d92a-4c15-8003-d3e8132297da/volumes" Oct 30 20:58:20.977249 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:20.977227 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d" path="/var/lib/kubelet/pods/f0e6a0bd-600e-46d0-b8c5-226aa8fbf72d/volumes" Oct 30 20:58:21.258113 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.257996 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx"] Oct 30 20:58:21.263196 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:58:21.263143 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598f3e3d_8117_4d80_b192_8fa7799a2420.slice/crio-d7b8929ba8540c0bd28626da9b64ebd9fa48f19df6f031ac8c18e466af9f6a89 WatchSource:0}: Error finding container d7b8929ba8540c0bd28626da9b64ebd9fa48f19df6f031ac8c18e466af9f6a89: Status 404 returned error can't find the container with id d7b8929ba8540c0bd28626da9b64ebd9fa48f19df6f031ac8c18e466af9f6a89 Oct 30 20:58:21.280255 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.280228 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd"] Oct 30 20:58:21.283237 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:58:21.283202 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d07b7cf_5e5f_470d_82d8_6ff84370fdf3.slice/crio-dc190bfbe738417d81dcfad9657703b5fe5eb4b58c382f7a6419b7059c7464a9 WatchSource:0}: Error finding container dc190bfbe738417d81dcfad9657703b5fe5eb4b58c382f7a6419b7059c7464a9: Status 404 returned error can't find the container with id dc190bfbe738417d81dcfad9657703b5fe5eb4b58c382f7a6419b7059c7464a9 Oct 30 20:58:21.334494 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.334454 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2"] Oct 30 20:58:21.344987 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.341350 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.349219 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.349102 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-70cf03-openshift-default-dockercfg-dfclm\"" Oct 30 20:58:21.351856 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.351794 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2"] Oct 30 20:58:21.403065 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.402988 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-podinfo\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.403251 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.403080 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istiod-ca-cert\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.403251 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.403123 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-envoy\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.403341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.403287 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-credential-socket\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.403421 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.403399 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-workload-socket\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.403523 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.403494 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-token\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.403588 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.403575 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-data\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.403675 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.403656 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-workload-certs\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.403740 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.403723 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7dh7\" (UniqueName: \"kubernetes.io/projected/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-kube-api-access-d7dh7\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.517840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.509738 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dh7\" (UniqueName: \"kubernetes.io/projected/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-kube-api-access-d7dh7\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.517840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.509852 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-podinfo\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.517840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.509895 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istiod-ca-cert\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.517840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.509941 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-envoy\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.517840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.510045 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-credential-socket\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.517840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.510142 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-workload-socket\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.517840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.510206 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-token\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.517840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.510276 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-data\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.517840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.510334 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-workload-certs\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.517840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.510938 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-workload-certs\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.517840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.511925 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-credential-socket\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.517840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.512745 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istiod-ca-cert\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.517840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.513642 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-workload-socket\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.517840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.513975 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-data\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.536880 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.536101 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-envoy\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.536880 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.536489 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-podinfo\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.573593 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.573493 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-token\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.574755 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.574646 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7dh7\" (UniqueName: \"kubernetes.io/projected/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-kube-api-access-d7dh7\") pod \"e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.692671 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.687199 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:21.988691 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.988653 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd" event={"ID":"5d07b7cf-5e5f-470d-82d8-6ff84370fdf3","Type":"ContainerStarted","Data":"ce19c38f0aabb2966bfb3c1379f5289c2d31a6485368318bf49a2b53b59f964a"} Oct 30 20:58:21.989133 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.988712 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd" event={"ID":"5d07b7cf-5e5f-470d-82d8-6ff84370fdf3","Type":"ContainerStarted","Data":"dc190bfbe738417d81dcfad9657703b5fe5eb4b58c382f7a6419b7059c7464a9"} Oct 30 20:58:21.989133 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.988744 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd" Oct 30 20:58:21.990335 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.990309 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx" event={"ID":"598f3e3d-8117-4d80-b192-8fa7799a2420","Type":"ContainerStarted","Data":"4beeaddbb23e1b435fd8cf0724cd71de506ef7265d5680fe8e8d0382a74867d1"} Oct 30 20:58:21.990441 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.990355 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx" event={"ID":"598f3e3d-8117-4d80-b192-8fa7799a2420","Type":"ContainerStarted","Data":"d7b8929ba8540c0bd28626da9b64ebd9fa48f19df6f031ac8c18e466af9f6a89"} Oct 30 20:58:21.990441 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:21.990394 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx" Oct 30 20:58:22.033295 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.033269 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2"] Oct 30 20:58:22.035711 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:58:22.035687 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eace1e0_9882_47ec_8c5d_927d8de6ffe6.slice/crio-1fd7b2a5cfe6e171fea9b6d0e7774e3fe4cc89b100a0811cbd6bc97339ed6760 WatchSource:0}: Error finding container 1fd7b2a5cfe6e171fea9b6d0e7774e3fe4cc89b100a0811cbd6bc97339ed6760: Status 404 returned error can't find the container with id 1fd7b2a5cfe6e171fea9b6d0e7774e3fe4cc89b100a0811cbd6bc97339ed6760 Oct 30 20:58:22.038667 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.038630 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:22.038756 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.038715 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:22.038812 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.038756 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:22.048808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.048737 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd" podStartSLOduration=2.048718942 podStartE2EDuration="2.048718942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:22.047116315 +0000 UTC m=+1207.227012823" watchObservedRunningTime="2026-10-30 20:58:22.048718942 +0000 UTC m=+1207.228615449" Oct 30 20:58:22.096169 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.096107 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx" podStartSLOduration=2.096087918 podStartE2EDuration="2.096087918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:22.091408027 +0000 UTC m=+1207.271304539" watchObservedRunningTime="2026-10-30 20:58:22.096087918 +0000 UTC m=+1207.275984425" Oct 30 20:58:22.114996 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.114967 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt"] Oct 30 20:58:22.123918 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.123892 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" Oct 30 20:58:22.139023 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.138994 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt"] Oct 30 20:58:22.139203 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.139186 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-c72dc9-serving-cert\"" Oct 30 20:58:22.225346 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.225297 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3b3cf05-d522-401f-81e2-3f221bc68b81-metrics-certs\") pod \"maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt\" (UID: \"c3b3cf05-d522-401f-81e2-3f221bc68b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" Oct 30 20:58:22.225510 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.225426 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3b3cf05-d522-401f-81e2-3f221bc68b81-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt\" (UID: \"c3b3cf05-d522-401f-81e2-3f221bc68b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" Oct 30 20:58:22.225510 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.225463 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b3cf05-d522-401f-81e2-3f221bc68b81-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt\" (UID: \"c3b3cf05-d522-401f-81e2-3f221bc68b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" Oct 30 20:58:22.225510 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.225500 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvkzh\" (UniqueName: \"kubernetes.io/projected/c3b3cf05-d522-401f-81e2-3f221bc68b81-kube-api-access-vvkzh\") pod \"maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt\" (UID: \"c3b3cf05-d522-401f-81e2-3f221bc68b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" Oct 30 20:58:22.324123 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.324038 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74"] Oct 30 20:58:22.330256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.330215 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3b3cf05-d522-401f-81e2-3f221bc68b81-metrics-certs\") pod \"maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt\" (UID: \"c3b3cf05-d522-401f-81e2-3f221bc68b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" Oct 30 20:58:22.330480 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.330349 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3b3cf05-d522-401f-81e2-3f221bc68b81-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt\" (UID: \"c3b3cf05-d522-401f-81e2-3f221bc68b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" Oct 30 20:58:22.330480 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.330387 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b3cf05-d522-401f-81e2-3f221bc68b81-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt\" (UID: \"c3b3cf05-d522-401f-81e2-3f221bc68b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" Oct 30 20:58:22.330480 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.330435 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvkzh\" (UniqueName: \"kubernetes.io/projected/c3b3cf05-d522-401f-81e2-3f221bc68b81-kube-api-access-vvkzh\") pod \"maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt\" (UID: \"c3b3cf05-d522-401f-81e2-3f221bc68b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" Oct 30 20:58:22.333220 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.333197 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.336308 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.336278 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b3cf05-d522-401f-81e2-3f221bc68b81-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt\" (UID: \"c3b3cf05-d522-401f-81e2-3f221bc68b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" Oct 30 20:58:22.336556 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.336533 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3b3cf05-d522-401f-81e2-3f221bc68b81-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt\" (UID: \"c3b3cf05-d522-401f-81e2-3f221bc68b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" Oct 30 20:58:22.342853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.338972 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74"] Oct 30 20:58:22.344817 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.344794 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-99b00f-openshift-default-dockercfg-qkww9\"" Oct 30 20:58:22.346459 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.346439 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3b3cf05-d522-401f-81e2-3f221bc68b81-metrics-certs\") pod \"maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt\" (UID: \"c3b3cf05-d522-401f-81e2-3f221bc68b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" Oct 30 20:58:22.359579 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.359551 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvkzh\" (UniqueName: \"kubernetes.io/projected/c3b3cf05-d522-401f-81e2-3f221bc68b81-kube-api-access-vvkzh\") pod \"maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt\" (UID: \"c3b3cf05-d522-401f-81e2-3f221bc68b81\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" Oct 30 20:58:22.432169 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.432135 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0998438-c7f6-4cb0-ba25-b406d43b8959-istiod-ca-cert\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.432169 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.432172 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-token\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.432454 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.432278 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-credential-socket\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.432454 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.432312 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-envoy\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.432454 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.432424 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-workload-certs\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.432584 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.432463 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-data\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.432584 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.432519 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-podinfo\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.432707 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.432693 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc287\" (UniqueName: \"kubernetes.io/projected/b0998438-c7f6-4cb0-ba25-b406d43b8959-kube-api-access-zc287\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.432803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.432716 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-workload-socket\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.444892 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.444865 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" Oct 30 20:58:22.538883 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.534665 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-workload-certs\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.538883 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.534935 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-data\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.538883 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.534986 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-podinfo\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.538883 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.535145 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc287\" (UniqueName: \"kubernetes.io/projected/b0998438-c7f6-4cb0-ba25-b406d43b8959-kube-api-access-zc287\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.538883 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.535173 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-workload-socket\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.538883 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.535309 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0998438-c7f6-4cb0-ba25-b406d43b8959-istiod-ca-cert\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.538883 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.535335 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-token\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.538883 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.535404 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-credential-socket\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.538883 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.535432 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-envoy\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.538883 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.536657 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-workload-certs\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.538883 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.536940 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-data\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.541606 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.540689 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0998438-c7f6-4cb0-ba25-b406d43b8959-istiod-ca-cert\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.541606 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.540996 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-workload-socket\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.541606 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.541554 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-credential-socket\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.602842 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.593373 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-token\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.614250 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.614169 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-envoy\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.622689 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.622609 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc287\" (UniqueName: \"kubernetes.io/projected/b0998438-c7f6-4cb0-ba25-b406d43b8959-kube-api-access-zc287\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.639182 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.637044 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-podinfo\") pod \"e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.705149 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:22.705103 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:22.726936 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:58:22.724688 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:58:23.008687 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.008474 4848 generic.go:356] "Generic (PLEG): container finished" podID="3a9a5360-6f89-4b28-9638-a56fdb01389f" containerID="8e520978851e1ebba7ed8eff2dc733057244d6c65c632e1b70d3a5db6b8c7036" exitCode=0 Oct 30 20:58:23.008687 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.008574 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" event={"ID":"3a9a5360-6f89-4b28-9638-a56fdb01389f","Type":"ContainerDied","Data":"8e520978851e1ebba7ed8eff2dc733057244d6c65c632e1b70d3a5db6b8c7036"} Oct 30 20:58:23.013125 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.012592 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" event={"ID":"2eace1e0-9882-47ec-8c5d-927d8de6ffe6","Type":"ContainerStarted","Data":"63ea6f5c00754eacadaeefa89d30b59385ea9c1909abfcbbb1d240648729a9ec"} Oct 30 20:58:23.013125 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.012655 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" event={"ID":"2eace1e0-9882-47ec-8c5d-927d8de6ffe6","Type":"ContainerStarted","Data":"1fd7b2a5cfe6e171fea9b6d0e7774e3fe4cc89b100a0811cbd6bc97339ed6760"} Oct 30 20:58:23.021855 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.021542 4848 generic.go:356] "Generic (PLEG): container finished" podID="932dfb46-276c-4d55-9800-f1cbaf96360d" containerID="cd3adba4e5182237fcf71de4636acd6c3e4f9f919ffef0e56881a5df452db327" exitCode=0 Oct 30 20:58:23.021855 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.021745 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" event={"ID":"932dfb46-276c-4d55-9800-f1cbaf96360d","Type":"ContainerDied","Data":"cd3adba4e5182237fcf71de4636acd6c3e4f9f919ffef0e56881a5df452db327"} Oct 30 20:58:23.025658 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.024890 4848 generic.go:356] "Generic (PLEG): container finished" podID="a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" containerID="6272471aa9cda5bc96709e1a2be742aa7e1b0a63ff6cf268b7c7add707e7f06f" exitCode=0 Oct 30 20:58:23.026134 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.026090 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" event={"ID":"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b","Type":"ContainerDied","Data":"6272471aa9cda5bc96709e1a2be742aa7e1b0a63ff6cf268b7c7add707e7f06f"} Oct 30 20:58:23.067666 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.067594 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" podStartSLOduration=2.067570402 podStartE2EDuration="2.067570402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:23.04333749 +0000 UTC m=+1208.223233999" watchObservedRunningTime="2026-10-30 20:58:23.067570402 +0000 UTC m=+1208.247466911" Oct 30 20:58:23.275416 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.275074 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt"] Oct 30 20:58:23.485794 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.485713 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:58:23.496471 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.496447 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:58:23.501701 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.501680 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:58:23.509934 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.509911 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74"] Oct 30 20:58:23.512795 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:58:23.511671 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0998438_c7f6_4cb0_ba25_b406d43b8959.slice/crio-fe31bc939c4943df7ab920b5cf7484cc268e291861a38d85366117943d4dd0ef WatchSource:0}: Error finding container fe31bc939c4943df7ab920b5cf7484cc268e291861a38d85366117943d4dd0ef: Status 404 returned error can't find the container with id fe31bc939c4943df7ab920b5cf7484cc268e291861a38d85366117943d4dd0ef Oct 30 20:58:23.518252 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.518220 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:23.518337 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.518285 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:23.518337 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.518317 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:23.636977 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.636937 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-workload-socket\") pod \"932dfb46-276c-4d55-9800-f1cbaf96360d\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " Oct 30 20:58:23.637186 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637006 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-token\" (UniqueName: \"kubernetes.io/projected/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-token\") pod \"932dfb46-276c-4d55-9800-f1cbaf96360d\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " Oct 30 20:58:23.637186 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637083 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-workload-socket\") pod \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " Oct 30 20:58:23.637186 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637176 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-credential-socket\") pod \"3a9a5360-6f89-4b28-9638-a56fdb01389f\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " Oct 30 20:58:23.637381 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637220 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-credential-socket\") pod \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " Oct 30 20:58:23.637381 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637280 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-credential-socket\") pod \"932dfb46-276c-4d55-9800-f1cbaf96360d\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " Oct 30 20:58:23.637381 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637313 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a9a5360-6f89-4b28-9638-a56fdb01389f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a9a5360-6f89-4b28-9638-a56fdb01389f-istiod-ca-cert\") pod \"3a9a5360-6f89-4b28-9638-a56fdb01389f\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " Oct 30 20:58:23.637381 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637353 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-token\" (UniqueName: \"kubernetes.io/projected/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-token\") pod \"3a9a5360-6f89-4b28-9638-a56fdb01389f\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " Oct 30 20:58:23.637584 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637419 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-podinfo\") pod \"3a9a5360-6f89-4b28-9638-a56fdb01389f\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " Oct 30 20:58:23.637584 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637453 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-workload-certs\") pod \"932dfb46-276c-4d55-9800-f1cbaf96360d\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " Oct 30 20:58:23.637584 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637483 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-data\") pod \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " Oct 30 20:58:23.637584 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637525 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932dfb46-276c-4d55-9800-f1cbaf96360d-kube-api-access-vpg9w\" (UniqueName: \"kubernetes.io/projected/932dfb46-276c-4d55-9800-f1cbaf96360d-kube-api-access-vpg9w\") pod \"932dfb46-276c-4d55-9800-f1cbaf96360d\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " Oct 30 20:58:23.637584 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637553 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-envoy\") pod \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " Oct 30 20:58:23.637853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637612 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a9a5360-6f89-4b28-9638-a56fdb01389f-kube-api-access-fmxd8\" (UniqueName: \"kubernetes.io/projected/3a9a5360-6f89-4b28-9638-a56fdb01389f-kube-api-access-fmxd8\") pod \"3a9a5360-6f89-4b28-9638-a56fdb01389f\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " Oct 30 20:58:23.637853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637643 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-data\") pod \"932dfb46-276c-4d55-9800-f1cbaf96360d\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " Oct 30 20:58:23.637853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637682 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-workload-socket\") pod \"3a9a5360-6f89-4b28-9638-a56fdb01389f\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " Oct 30 20:58:23.637853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637707 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-token\" (UniqueName: \"kubernetes.io/projected/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-token\") pod \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " Oct 30 20:58:23.637853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637747 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-podinfo\") pod \"932dfb46-276c-4d55-9800-f1cbaf96360d\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " Oct 30 20:58:23.637853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637829 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-kube-api-access-jggb7\" (UniqueName: \"kubernetes.io/projected/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-kube-api-access-jggb7\") pod \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " Oct 30 20:58:23.638152 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637877 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/932dfb46-276c-4d55-9800-f1cbaf96360d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932dfb46-276c-4d55-9800-f1cbaf96360d-istiod-ca-cert\") pod \"932dfb46-276c-4d55-9800-f1cbaf96360d\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " Oct 30 20:58:23.638152 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637911 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-workload-certs\") pod \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " Oct 30 20:58:23.638152 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637949 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-envoy\") pod \"932dfb46-276c-4d55-9800-f1cbaf96360d\" (UID: \"932dfb46-276c-4d55-9800-f1cbaf96360d\") " Oct 30 20:58:23.638152 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.637998 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istiod-ca-cert\") pod \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " Oct 30 20:58:23.638152 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.638047 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-workload-certs\") pod \"3a9a5360-6f89-4b28-9638-a56fdb01389f\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " Oct 30 20:58:23.638152 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.638109 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-podinfo\") pod \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\" (UID: \"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b\") " Oct 30 20:58:23.638152 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.638144 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-envoy\") pod \"3a9a5360-6f89-4b28-9638-a56fdb01389f\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " Oct 30 20:58:23.638496 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.638196 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-data\") pod \"3a9a5360-6f89-4b28-9638-a56fdb01389f\" (UID: \"3a9a5360-6f89-4b28-9638-a56fdb01389f\") " Oct 30 20:58:23.638496 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.638246 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-credential-socket" pod "932dfb46-276c-4d55-9800-f1cbaf96360d" (UID: "932dfb46-276c-4d55-9800-f1cbaf96360d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:23.638496 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.638349 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-credential-socket" pod "3a9a5360-6f89-4b28-9638-a56fdb01389f" (UID: "3a9a5360-6f89-4b28-9638-a56fdb01389f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:23.638496 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.638410 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-workload-socket" pod "a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" (UID: "a18d4981-f49d-4f1a-b8a3-c0fb3248c72b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:23.638697 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.638612 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-data" pod "3a9a5360-6f89-4b28-9638-a56fdb01389f" (UID: "3a9a5360-6f89-4b28-9638-a56fdb01389f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:23.639755 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.638735 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-credential-socket" pod "a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" (UID: "a18d4981-f49d-4f1a-b8a3-c0fb3248c72b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:23.639755 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.638895 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a9a5360-6f89-4b28-9638-a56fdb01389f-istiod-ca-cert" pod "3a9a5360-6f89-4b28-9638-a56fdb01389f" (UID: "3a9a5360-6f89-4b28-9638-a56fdb01389f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:23.639755 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.638999 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.639755 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.639096 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.639755 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.639111 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.639755 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.639125 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.639755 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.639140 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a9a5360-6f89-4b28-9638-a56fdb01389f-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.639755 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.639154 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.639755 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.639193 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-workload-socket" pod "932dfb46-276c-4d55-9800-f1cbaf96360d" (UID: "932dfb46-276c-4d55-9800-f1cbaf96360d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:23.639755 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.639448 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-data" pod "932dfb46-276c-4d55-9800-f1cbaf96360d" (UID: "932dfb46-276c-4d55-9800-f1cbaf96360d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:23.639755 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.639495 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932dfb46-276c-4d55-9800-f1cbaf96360d-istiod-ca-cert" pod "932dfb46-276c-4d55-9800-f1cbaf96360d" (UID: "932dfb46-276c-4d55-9800-f1cbaf96360d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:23.639755 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.639756 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-workload-certs" pod "a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" (UID: "a18d4981-f49d-4f1a-b8a3-c0fb3248c72b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:23.641049 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.641016 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-token" pod "932dfb46-276c-4d55-9800-f1cbaf96360d" (UID: "932dfb46-276c-4d55-9800-f1cbaf96360d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:23.641271 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.641246 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-data" pod "a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" (UID: "a18d4981-f49d-4f1a-b8a3-c0fb3248c72b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:23.641408 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.641375 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-workload-socket" pod "3a9a5360-6f89-4b28-9638-a56fdb01389f" (UID: "3a9a5360-6f89-4b28-9638-a56fdb01389f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:23.641515 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.641493 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-workload-certs" pod "932dfb46-276c-4d55-9800-f1cbaf96360d" (UID: "932dfb46-276c-4d55-9800-f1cbaf96360d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:23.641820 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.641756 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-workload-certs" pod "3a9a5360-6f89-4b28-9638-a56fdb01389f" (UID: "3a9a5360-6f89-4b28-9638-a56fdb01389f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:23.641932 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.641895 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-token" pod "3a9a5360-6f89-4b28-9638-a56fdb01389f" (UID: "3a9a5360-6f89-4b28-9638-a56fdb01389f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:23.642414 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.642382 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istiod-ca-cert" pod "a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" (UID: "a18d4981-f49d-4f1a-b8a3-c0fb3248c72b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:23.643014 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.642989 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-envoy" pod "932dfb46-276c-4d55-9800-f1cbaf96360d" (UID: "932dfb46-276c-4d55-9800-f1cbaf96360d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:23.643105 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.643073 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-kube-api-access-jggb7" pod "a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" (UID: "a18d4981-f49d-4f1a-b8a3-c0fb3248c72b"). InnerVolumeSpecName "kube-api-access-jggb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:23.643881 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.643856 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-podinfo" pod "932dfb46-276c-4d55-9800-f1cbaf96360d" (UID: "932dfb46-276c-4d55-9800-f1cbaf96360d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:23.643971 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.643959 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-podinfo" pod "3a9a5360-6f89-4b28-9638-a56fdb01389f" (UID: "3a9a5360-6f89-4b28-9638-a56fdb01389f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:23.644278 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.644183 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-envoy" pod "a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" (UID: "a18d4981-f49d-4f1a-b8a3-c0fb3248c72b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:23.644278 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.644243 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-podinfo" pod "a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" (UID: "a18d4981-f49d-4f1a-b8a3-c0fb3248c72b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:23.645608 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.645576 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-token" pod "a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" (UID: "a18d4981-f49d-4f1a-b8a3-c0fb3248c72b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:23.645753 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.645732 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-envoy" pod "3a9a5360-6f89-4b28-9638-a56fdb01389f" (UID: "3a9a5360-6f89-4b28-9638-a56fdb01389f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:23.645876 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.645831 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9a5360-6f89-4b28-9638-a56fdb01389f-kube-api-access-fmxd8" pod "3a9a5360-6f89-4b28-9638-a56fdb01389f" (UID: "3a9a5360-6f89-4b28-9638-a56fdb01389f"). InnerVolumeSpecName "kube-api-access-fmxd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:23.646090 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.646062 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932dfb46-276c-4d55-9800-f1cbaf96360d-kube-api-access-vpg9w" pod "932dfb46-276c-4d55-9800-f1cbaf96360d" (UID: "932dfb46-276c-4d55-9800-f1cbaf96360d"). InnerVolumeSpecName "kube-api-access-vpg9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:23.692363 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.690786 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:23.699342 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.699308 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:23.741209 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741133 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.741209 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741169 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jggb7\" (UniqueName: \"kubernetes.io/projected/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-kube-api-access-jggb7\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.741209 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741186 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932dfb46-276c-4d55-9800-f1cbaf96360d-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.741209 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741200 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.757180 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741218 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.757180 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741232 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.757180 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741246 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.757180 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741256 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.757180 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741268 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.757180 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741281 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.757180 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741294 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.757180 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741308 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.757180 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741321 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a9a5360-6f89-4b28-9638-a56fdb01389f-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.757180 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741335 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.757180 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741348 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.757180 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741363 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpg9w\" (UniqueName: \"kubernetes.io/projected/932dfb46-276c-4d55-9800-f1cbaf96360d-kube-api-access-vpg9w\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.757180 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741371 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.757180 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741381 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmxd8\" (UniqueName: \"kubernetes.io/projected/3a9a5360-6f89-4b28-9638-a56fdb01389f-kube-api-access-fmxd8\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.757180 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741390 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932dfb46-276c-4d55-9800-f1cbaf96360d-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.757180 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741404 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a9a5360-6f89-4b28-9638-a56fdb01389f-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:23.757180 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:23.741418 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:24.033837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.033704 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" event={"ID":"3a9a5360-6f89-4b28-9638-a56fdb01389f","Type":"ContainerDied","Data":"79fb15329e6e69cd09a8f64b1964e1e064779d7eb2caf99c4533c6c61ee190e8"} Oct 30 20:58:24.033837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.033714 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz" Oct 30 20:58:24.033837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.033783 4848 scope.go:122] "RemoveContainer" containerID="8e520978851e1ebba7ed8eff2dc733057244d6c65c632e1b70d3a5db6b8c7036" Oct 30 20:58:24.035496 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.035477 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" Oct 30 20:58:24.035622 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.035495 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd" event={"ID":"932dfb46-276c-4d55-9800-f1cbaf96360d","Type":"ContainerDied","Data":"87fbf9340a4a7401810cd185143825fe0b3c5d0d16b3459b0b728e2fe4aa9ad9"} Oct 30 20:58:24.037673 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.037487 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" event={"ID":"c3b3cf05-d522-401f-81e2-3f221bc68b81","Type":"ContainerStarted","Data":"db7745141ad7c0ea45a9b0658d9a5c2478daa8c2347d60f83fea495cd8faaeae"} Oct 30 20:58:24.037673 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.037540 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" event={"ID":"c3b3cf05-d522-401f-81e2-3f221bc68b81","Type":"ContainerStarted","Data":"1d32b12fbd83b4ed314a9b23395c99fd6b2f6098730fe1b8725139f67563fea7"} Oct 30 20:58:24.037673 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.037571 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" Oct 30 20:58:24.039103 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.039083 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" Oct 30 20:58:24.039209 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.039101 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j" event={"ID":"a18d4981-f49d-4f1a-b8a3-c0fb3248c72b","Type":"ContainerDied","Data":"7acac36e150209a5658f26cbd7016990cdfd1272fd11669f90ec62ba9cbde213"} Oct 30 20:58:24.041059 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.041037 4848 generic.go:356] "Generic (PLEG): container finished" podID="46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" containerID="9fcb9d3e9c7a735ea417b94694a721a519405bdeb6be2be0ad24e54d188492f5" exitCode=0 Oct 30 20:58:24.041179 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.041162 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" event={"ID":"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80","Type":"ContainerDied","Data":"9fcb9d3e9c7a735ea417b94694a721a519405bdeb6be2be0ad24e54d188492f5"} Oct 30 20:58:24.041525 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.041508 4848 scope.go:122] "RemoveContainer" containerID="9fcb9d3e9c7a735ea417b94694a721a519405bdeb6be2be0ad24e54d188492f5" Oct 30 20:58:24.043977 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.043956 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" event={"ID":"b0998438-c7f6-4cb0-ba25-b406d43b8959","Type":"ContainerStarted","Data":"65b26dc1623be9f3195ac4dbca57546a9696e367a27f16f7eb8099116c3df05d"} Oct 30 20:58:24.044381 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.043998 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" event={"ID":"b0998438-c7f6-4cb0-ba25-b406d43b8959","Type":"ContainerStarted","Data":"fe31bc939c4943df7ab920b5cf7484cc268e291861a38d85366117943d4dd0ef"} Oct 30 20:58:24.044543 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.044526 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:24.047318 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.047299 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:58:24.048749 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.048718 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:24.049016 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.048843 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:24.049016 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.048891 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:24.049016 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.048984 4848 scope.go:122] "RemoveContainer" containerID="cd3adba4e5182237fcf71de4636acd6c3e4f9f919ffef0e56881a5df452db327" Oct 30 20:58:24.057275 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.057235 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" podStartSLOduration=1.543958037 podStartE2EDuration="2.057222905s" totalImagesPullingTime="513.264868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:24.056959737 +0000 UTC m=+1209.236856246" watchObservedRunningTime="2026-10-30 20:58:24.057222905 +0000 UTC m=+1209.237119412" Oct 30 20:58:24.074794 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.074746 4848 scope.go:122] "RemoveContainer" containerID="6272471aa9cda5bc96709e1a2be742aa7e1b0a63ff6cf268b7c7add707e7f06f" Oct 30 20:58:24.094031 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.093992 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" podStartSLOduration=2.093978045 podStartE2EDuration="2.093978045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:24.091356634 +0000 UTC m=+1209.271253168" watchObservedRunningTime="2026-10-30 20:58:24.093978045 +0000 UTC m=+1209.273874552" Oct 30 20:58:24.151433 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.151408 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz"] Oct 30 20:58:24.157328 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.157211 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-pl2cz"] Oct 30 20:58:24.162069 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.162015 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j"] Oct 30 20:58:24.165219 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.165195 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-xrf6j"] Oct 30 20:58:24.222157 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.222097 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd"] Oct 30 20:58:24.227645 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.227605 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-t8zxd"] Oct 30 20:58:24.705735 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.705697 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:24.711059 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.711036 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:24.943711 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.943676 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a9a5360-6f89-4b28-9638-a56fdb01389f" path="/var/lib/kubelet/pods/3a9a5360-6f89-4b28-9638-a56fdb01389f/volumes" Oct 30 20:58:24.944267 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.944250 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="932dfb46-276c-4d55-9800-f1cbaf96360d" path="/var/lib/kubelet/pods/932dfb46-276c-4d55-9800-f1cbaf96360d/volumes" Oct 30 20:58:24.944690 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:24.944676 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" path="/var/lib/kubelet/pods/a18d4981-f49d-4f1a-b8a3-c0fb3248c72b/volumes" Oct 30 20:58:25.054440 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:25.054350 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" event={"ID":"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80","Type":"ContainerStarted","Data":"d59ebc8b5ee90d2d693830b89d3c4a4bcf731592013ef870d811c31ab0773cca"} Oct 30 20:58:25.056900 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:25.056874 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:25.057641 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:25.057622 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:58:25.823509 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:25.823472 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:58:25.823719 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:25.823593 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 20:58:25.823719 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:25.823635 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" podUID="46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 20:58:26.695630 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:26.695603 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh" Oct 30 20:58:26.696072 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:26.695667 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz" Oct 30 20:58:26.804126 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:26.804093 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 20:58:26.821868 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:26.821826 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" podUID="46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 20:58:27.194606 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.194459 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr"] Oct 30 20:58:27.196845 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.195996 4848 cpu_manager.go:414] "containerMap: removing container" podUID="932dfb46-276c-4d55-9800-f1cbaf96360d" containerName="istio-proxy" Oct 30 20:58:27.197040 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.197022 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932dfb46-276c-4d55-9800-f1cbaf96360d" containerName="istio-proxy" Oct 30 20:58:27.197170 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.197149 4848 cpu_manager.go:414] "containerMap: removing container" podUID="3a9a5360-6f89-4b28-9638-a56fdb01389f" containerName="istio-proxy" Oct 30 20:58:27.197256 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.197244 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a9a5360-6f89-4b28-9638-a56fdb01389f" containerName="istio-proxy" Oct 30 20:58:27.197370 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.197360 4848 cpu_manager.go:414] "containerMap: removing container" podUID="a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" containerName="istio-proxy" Oct 30 20:58:27.197446 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.197435 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" containerName="istio-proxy" Oct 30 20:58:27.197685 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.197672 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a9a5360-6f89-4b28-9638-a56fdb01389f" containerName="istio-proxy" Oct 30 20:58:27.197807 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.197796 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18d4981-f49d-4f1a-b8a3-c0fb3248c72b" containerName="istio-proxy" Oct 30 20:58:27.197913 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.197903 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="932dfb46-276c-4d55-9800-f1cbaf96360d" containerName="istio-proxy" Oct 30 20:58:27.268831 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.265888 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr"] Oct 30 20:58:27.268831 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.266027 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r"] Oct 30 20:58:27.268831 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.267964 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr" Oct 30 20:58:27.276171 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.276142 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-70cf03-dockercfg-x2mt2\"" Oct 30 20:58:27.299888 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.277095 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-70cf03\"" Oct 30 20:58:27.300110 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.283389 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r"] Oct 30 20:58:27.300291 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.283551 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r" Oct 30 20:58:27.443835 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.443793 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/168fbd96-d5d2-43ea-b18e-26cd242b53cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr\" (UID: \"168fbd96-d5d2-43ea-b18e-26cd242b53cb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr" Oct 30 20:58:27.452990 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.452907 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4j4t\" (UniqueName: \"kubernetes.io/projected/168fbd96-d5d2-43ea-b18e-26cd242b53cb-kube-api-access-h4j4t\") pod \"payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr\" (UID: \"168fbd96-d5d2-43ea-b18e-26cd242b53cb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr" Oct 30 20:58:27.453289 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.453268 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc891d1-c6b8-4650-bd1a-e14115e157bb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r\" (UID: \"6cc891d1-c6b8-4650-bd1a-e14115e157bb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r" Oct 30 20:58:27.471788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.467949 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x4gz\" (UniqueName: \"kubernetes.io/projected/6cc891d1-c6b8-4650-bd1a-e14115e157bb-kube-api-access-8x4gz\") pod \"payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r\" (UID: \"6cc891d1-c6b8-4650-bd1a-e14115e157bb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r" Oct 30 20:58:27.575742 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.575698 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/168fbd96-d5d2-43ea-b18e-26cd242b53cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr\" (UID: \"168fbd96-d5d2-43ea-b18e-26cd242b53cb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr" Oct 30 20:58:27.576067 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.576038 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4j4t\" (UniqueName: \"kubernetes.io/projected/168fbd96-d5d2-43ea-b18e-26cd242b53cb-kube-api-access-h4j4t\") pod \"payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr\" (UID: \"168fbd96-d5d2-43ea-b18e-26cd242b53cb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr" Oct 30 20:58:27.576218 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.576202 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc891d1-c6b8-4650-bd1a-e14115e157bb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r\" (UID: \"6cc891d1-c6b8-4650-bd1a-e14115e157bb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r" Oct 30 20:58:27.576353 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.576326 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x4gz\" (UniqueName: \"kubernetes.io/projected/6cc891d1-c6b8-4650-bd1a-e14115e157bb-kube-api-access-8x4gz\") pod \"payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r\" (UID: \"6cc891d1-c6b8-4650-bd1a-e14115e157bb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r" Oct 30 20:58:27.577709 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.577686 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/168fbd96-d5d2-43ea-b18e-26cd242b53cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr\" (UID: \"168fbd96-d5d2-43ea-b18e-26cd242b53cb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr" Oct 30 20:58:27.577933 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.577824 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc891d1-c6b8-4650-bd1a-e14115e157bb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r\" (UID: \"6cc891d1-c6b8-4650-bd1a-e14115e157bb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r" Oct 30 20:58:27.600911 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.600093 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x4gz\" (UniqueName: \"kubernetes.io/projected/6cc891d1-c6b8-4650-bd1a-e14115e157bb-kube-api-access-8x4gz\") pod \"payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r\" (UID: \"6cc891d1-c6b8-4650-bd1a-e14115e157bb\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r" Oct 30 20:58:27.602672 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.602647 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4j4t\" (UniqueName: \"kubernetes.io/projected/168fbd96-d5d2-43ea-b18e-26cd242b53cb-kube-api-access-h4j4t\") pod \"payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr\" (UID: \"168fbd96-d5d2-43ea-b18e-26cd242b53cb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr" Oct 30 20:58:27.695382 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.695353 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr" Oct 30 20:58:27.782201 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.782172 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r" Oct 30 20:58:27.845783 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.832090 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 20:58:27.845783 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:27.832153 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" podUID="46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 20:58:28.096404 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:58:28.096372 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168fbd96_d5d2_43ea_b18e_26cd242b53cb.slice/crio-b4fe09970f1f258c5da6c98c4602545c682b822cf159a4980a9b1a26f0822170 WatchSource:0}: Error finding container b4fe09970f1f258c5da6c98c4602545c682b822cf159a4980a9b1a26f0822170: Status 404 returned error can't find the container with id b4fe09970f1f258c5da6c98c4602545c682b822cf159a4980a9b1a26f0822170 Oct 30 20:58:28.098196 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:28.098172 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr"] Oct 30 20:58:28.162695 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:28.162240 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr" event={"ID":"168fbd96-d5d2-43ea-b18e-26cd242b53cb","Type":"ContainerStarted","Data":"b4fe09970f1f258c5da6c98c4602545c682b822cf159a4980a9b1a26f0822170"} Oct 30 20:58:28.191536 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:28.191508 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r"] Oct 30 20:58:28.194042 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:58:28.194017 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc891d1_c6b8_4650_bd1a_e14115e157bb.slice/crio-c1079141147045247d2653c6630a28576ef36b0069c002a85a35a465c9dfa5c8 WatchSource:0}: Error finding container c1079141147045247d2653c6630a28576ef36b0069c002a85a35a465c9dfa5c8: Status 404 returned error can't find the container with id c1079141147045247d2653c6630a28576ef36b0069c002a85a35a465c9dfa5c8 Oct 30 20:58:28.775347 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:28.775317 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9"] Oct 30 20:58:28.775813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:28.775780 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" podUID="2bdeff6d-2976-4534-8857-b04b966de5c4" containerName="istio-proxy" containerID="cri-o://93320dffc87b4a3d3e0c23dfdc27b402462a45a1eee0a23a9aa450c8e4b64243" gracePeriod=30 Oct 30 20:58:28.778810 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:28.778785 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8"] Oct 30 20:58:28.779021 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:28.778997 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" podUID="0dafd76f-6a6c-479f-86e3-b67f973cab0e" containerName="istio-proxy" containerID="cri-o://7c370286b08a8341526e2c1122ab46701e35259978fd2b6f7207cf8cba5cc34b" gracePeriod=30 Oct 30 20:58:28.803234 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:28.803202 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 20:58:28.803600 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:28.803256 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" podUID="46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 20:58:28.890967 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:28.888961 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw"] Oct 30 20:58:28.894325 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:28.894022 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw"] Oct 30 20:58:28.894325 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:28.894067 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt"] Oct 30 20:58:28.897354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:28.897333 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw" Oct 30 20:58:28.901651 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:28.901556 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt"] Oct 30 20:58:28.903713 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:28.901903 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt" Oct 30 20:58:28.933054 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:28.902867 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-99b00f\"" Oct 30 20:58:29.056107 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:28.903009 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-99b00f-dockercfg-wn74x\"" Oct 30 20:58:29.331848 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.285318 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkg69\" (UniqueName: \"kubernetes.io/projected/ce16f073-8c67-4080-aa8d-3882ac35fc6c-kube-api-access-pkg69\") pod \"payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw\" (UID: \"ce16f073-8c67-4080-aa8d-3882ac35fc6c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw" Oct 30 20:58:29.371293 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.370967 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce16f073-8c67-4080-aa8d-3882ac35fc6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw\" (UID: \"ce16f073-8c67-4080-aa8d-3882ac35fc6c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw" Oct 30 20:58:29.371293 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.371046 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477e7c99-1a03-4ea9-b4e3-97b7488ef355-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt\" (UID: \"477e7c99-1a03-4ea9-b4e3-97b7488ef355\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt" Oct 30 20:58:29.371293 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.371189 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ww5l\" (UniqueName: \"kubernetes.io/projected/477e7c99-1a03-4ea9-b4e3-97b7488ef355-kube-api-access-9ww5l\") pod \"payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt\" (UID: \"477e7c99-1a03-4ea9-b4e3-97b7488ef355\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt" Oct 30 20:58:29.400246 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.400207 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r" event={"ID":"6cc891d1-c6b8-4650-bd1a-e14115e157bb","Type":"ContainerStarted","Data":"d67dd5a5ce1abd8e2fe99a53f6010e3f91267eda37215b1a8c8a448866cf6f27"} Oct 30 20:58:29.400389 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.400271 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r" event={"ID":"6cc891d1-c6b8-4650-bd1a-e14115e157bb","Type":"ContainerStarted","Data":"c1079141147045247d2653c6630a28576ef36b0069c002a85a35a465c9dfa5c8"} Oct 30 20:58:29.400389 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.400314 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r" Oct 30 20:58:29.401976 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.401952 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr" event={"ID":"168fbd96-d5d2-43ea-b18e-26cd242b53cb","Type":"ContainerStarted","Data":"fba97c65d09fe9999762734a3f1900d27f7bf2d4f039c45d8a5d59950776d8b9"} Oct 30 20:58:29.402075 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.401999 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr" Oct 30 20:58:29.453432 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.453375 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r" podStartSLOduration=2.453355183 podStartE2EDuration="2.453355183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:29.448568493 +0000 UTC m=+1214.628465001" watchObservedRunningTime="2026-10-30 20:58:29.453355183 +0000 UTC m=+1214.633251690" Oct 30 20:58:29.474618 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.474561 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr" podStartSLOduration=2.4745442300000002 podStartE2EDuration="2.47454423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:29.469817071 +0000 UTC m=+1214.649713593" watchObservedRunningTime="2026-10-30 20:58:29.47454423 +0000 UTC m=+1214.654440827" Oct 30 20:58:29.474841 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.474678 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkg69\" (UniqueName: \"kubernetes.io/projected/ce16f073-8c67-4080-aa8d-3882ac35fc6c-kube-api-access-pkg69\") pod \"payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw\" (UID: \"ce16f073-8c67-4080-aa8d-3882ac35fc6c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw" Oct 30 20:58:29.474841 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.474749 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce16f073-8c67-4080-aa8d-3882ac35fc6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw\" (UID: \"ce16f073-8c67-4080-aa8d-3882ac35fc6c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw" Oct 30 20:58:29.474841 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.474826 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477e7c99-1a03-4ea9-b4e3-97b7488ef355-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt\" (UID: \"477e7c99-1a03-4ea9-b4e3-97b7488ef355\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt" Oct 30 20:58:29.474967 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.474868 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ww5l\" (UniqueName: \"kubernetes.io/projected/477e7c99-1a03-4ea9-b4e3-97b7488ef355-kube-api-access-9ww5l\") pod \"payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt\" (UID: \"477e7c99-1a03-4ea9-b4e3-97b7488ef355\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt" Oct 30 20:58:29.475728 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.475709 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce16f073-8c67-4080-aa8d-3882ac35fc6c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw\" (UID: \"ce16f073-8c67-4080-aa8d-3882ac35fc6c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw" Oct 30 20:58:29.475805 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.475712 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477e7c99-1a03-4ea9-b4e3-97b7488ef355-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt\" (UID: \"477e7c99-1a03-4ea9-b4e3-97b7488ef355\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt" Oct 30 20:58:29.484523 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.484494 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkg69\" (UniqueName: \"kubernetes.io/projected/ce16f073-8c67-4080-aa8d-3882ac35fc6c-kube-api-access-pkg69\") pod \"payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw\" (UID: \"ce16f073-8c67-4080-aa8d-3882ac35fc6c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw" Oct 30 20:58:29.484663 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.484583 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ww5l\" (UniqueName: \"kubernetes.io/projected/477e7c99-1a03-4ea9-b4e3-97b7488ef355-kube-api-access-9ww5l\") pod \"payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt\" (UID: \"477e7c99-1a03-4ea9-b4e3-97b7488ef355\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt" Oct 30 20:58:29.560366 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.560320 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw" Oct 30 20:58:29.711790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.705394 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt" Oct 30 20:58:29.861837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.803350 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 20:58:29.861837 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:29.803408 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" podUID="46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 20:58:30.076941 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.076642 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f"] Oct 30 20:58:30.087214 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.087170 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" Oct 30 20:58:30.087275 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.087216 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw"] Oct 30 20:58:30.087409 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.087385 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" Oct 30 20:58:30.090406 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.090380 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f"] Oct 30 20:58:30.091364 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.091346 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-955eb7-serving-cert\"" Oct 30 20:58:30.200958 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.200926 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt"] Oct 30 20:58:30.202119 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.202092 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e80f4869-c21f-4f08-b997-ac9c56c23619-metrics-certs\") pod \"maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f\" (UID: \"e80f4869-c21f-4f08-b997-ac9c56c23619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" Oct 30 20:58:30.202270 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:58:30.202235 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477e7c99_1a03_4ea9_b4e3_97b7488ef355.slice/crio-41ce7462c9a4839811a32b781f0b26266774c02f0bc9fed5450114330443f806 WatchSource:0}: Error finding container 41ce7462c9a4839811a32b781f0b26266774c02f0bc9fed5450114330443f806: Status 404 returned error can't find the container with id 41ce7462c9a4839811a32b781f0b26266774c02f0bc9fed5450114330443f806 Oct 30 20:58:30.202808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.202457 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80f4869-c21f-4f08-b997-ac9c56c23619-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f\" (UID: \"e80f4869-c21f-4f08-b997-ac9c56c23619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" Oct 30 20:58:30.202808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.202618 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e80f4869-c21f-4f08-b997-ac9c56c23619-maas-api-tls\") pod \"maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f\" (UID: \"e80f4869-c21f-4f08-b997-ac9c56c23619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" Oct 30 20:58:30.202808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.202709 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c46kw\" (UniqueName: \"kubernetes.io/projected/e80f4869-c21f-4f08-b997-ac9c56c23619-kube-api-access-c46kw\") pod \"maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f\" (UID: \"e80f4869-c21f-4f08-b997-ac9c56c23619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" Oct 30 20:58:30.305146 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.304727 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e80f4869-c21f-4f08-b997-ac9c56c23619-maas-api-tls\") pod \"maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f\" (UID: \"e80f4869-c21f-4f08-b997-ac9c56c23619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" Oct 30 20:58:30.305146 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.304837 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c46kw\" (UniqueName: \"kubernetes.io/projected/e80f4869-c21f-4f08-b997-ac9c56c23619-kube-api-access-c46kw\") pod \"maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f\" (UID: \"e80f4869-c21f-4f08-b997-ac9c56c23619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" Oct 30 20:58:30.305146 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.304887 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e80f4869-c21f-4f08-b997-ac9c56c23619-metrics-certs\") pod \"maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f\" (UID: \"e80f4869-c21f-4f08-b997-ac9c56c23619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" Oct 30 20:58:30.305146 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.305072 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80f4869-c21f-4f08-b997-ac9c56c23619-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f\" (UID: \"e80f4869-c21f-4f08-b997-ac9c56c23619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" Oct 30 20:58:30.305945 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.305917 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80f4869-c21f-4f08-b997-ac9c56c23619-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f\" (UID: \"e80f4869-c21f-4f08-b997-ac9c56c23619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" Oct 30 20:58:30.400067 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.399473 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c46kw\" (UniqueName: \"kubernetes.io/projected/e80f4869-c21f-4f08-b997-ac9c56c23619-kube-api-access-c46kw\") pod \"maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f\" (UID: \"e80f4869-c21f-4f08-b997-ac9c56c23619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" Oct 30 20:58:30.403522 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.403461 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e80f4869-c21f-4f08-b997-ac9c56c23619-metrics-certs\") pod \"maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f\" (UID: \"e80f4869-c21f-4f08-b997-ac9c56c23619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" Oct 30 20:58:30.414014 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.407416 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e80f4869-c21f-4f08-b997-ac9c56c23619-maas-api-tls\") pod \"maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f\" (UID: \"e80f4869-c21f-4f08-b997-ac9c56c23619\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" Oct 30 20:58:30.469146 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.469109 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt" event={"ID":"477e7c99-1a03-4ea9-b4e3-97b7488ef355","Type":"ContainerStarted","Data":"41ce7462c9a4839811a32b781f0b26266774c02f0bc9fed5450114330443f806"} Oct 30 20:58:30.472662 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.472639 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw" event={"ID":"ce16f073-8c67-4080-aa8d-3882ac35fc6c","Type":"ContainerStarted","Data":"56fa7c02f53cff1512f0dc1eed8d2e8682fabaf57e05d76f3358fac34544c359"} Oct 30 20:58:30.481316 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.481265 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" Oct 30 20:58:30.852841 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.849841 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 20:58:30.852841 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:30.849923 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" podUID="46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 20:58:31.078949 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:31.072326 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f"] Oct 30 20:58:31.487142 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:31.487103 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt" event={"ID":"477e7c99-1a03-4ea9-b4e3-97b7488ef355","Type":"ContainerStarted","Data":"64c9d95316a498b9ee8ed7d5101965d5e9b9068300e69d8fdad31b5d13f7fdb9"} Oct 30 20:58:31.487347 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:31.487197 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt" Oct 30 20:58:31.490825 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:31.490757 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw" event={"ID":"ce16f073-8c67-4080-aa8d-3882ac35fc6c","Type":"ContainerStarted","Data":"f716e06319b2232a0a3e0761ddb462e4524dd6ae5517acaad19488226bb7580d"} Oct 30 20:58:31.490959 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:31.490949 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw" Oct 30 20:58:31.492194 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:31.492171 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" event={"ID":"e80f4869-c21f-4f08-b997-ac9c56c23619","Type":"ContainerStarted","Data":"f5fcd22882d1774db7681b5f829b761d53e3324dbae2da3956cbc0f949d8f039"} Oct 30 20:58:31.527803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:31.523780 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt" podStartSLOduration=3.523743153 podStartE2EDuration="3.523743153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:31.518227329 +0000 UTC m=+1216.698123837" watchObservedRunningTime="2026-10-30 20:58:31.523743153 +0000 UTC m=+1216.703639661" Oct 30 20:58:31.544517 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:31.544447 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw" podStartSLOduration=3.54442745 podStartE2EDuration="3.54442745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:31.54103704 +0000 UTC m=+1216.720933548" watchObservedRunningTime="2026-10-30 20:58:31.54442745 +0000 UTC m=+1216.724323959" Oct 30 20:58:31.835844 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:31.835811 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:58:31.837048 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:31.837026 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:58:31.838830 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:31.838811 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:58:32.500984 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:32.500950 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" event={"ID":"e80f4869-c21f-4f08-b997-ac9c56c23619","Type":"ContainerStarted","Data":"1d812e26831edce3742eb53279c186269f880050b43c07b899cdd0f4b63ec083"} Oct 30 20:58:32.501412 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:32.501153 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" Oct 30 20:58:32.532575 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:32.532515 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" podStartSLOduration=2.992064714 podStartE2EDuration="3.532501116s" totalImagesPullingTime="540.436402ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:32.529690862 +0000 UTC m=+1217.709587370" watchObservedRunningTime="2026-10-30 20:58:32.532501116 +0000 UTC m=+1217.712397622" Oct 30 20:58:33.030115 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:33.030085 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx" Oct 30 20:58:33.030382 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:33.030364 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd" Oct 30 20:58:34.102394 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.102024 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:58:34.141608 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.141556 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:58:34.146263 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.146231 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-workload-socket\") pod \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " Oct 30 20:58:34.146434 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.146272 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-workload-certs\") pod \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " Oct 30 20:58:34.146434 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.146320 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dafd76f-6a6c-479f-86e3-b67f973cab0e-kube-api-access-rml9s\" (UniqueName: \"kubernetes.io/projected/0dafd76f-6a6c-479f-86e3-b67f973cab0e-kube-api-access-rml9s\") pod \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " Oct 30 20:58:34.146434 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.146424 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-data\") pod \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " Oct 30 20:58:34.146589 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.146456 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-credential-socket\") pod \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " Oct 30 20:58:34.146589 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.146536 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-workload-socket" pod "0dafd76f-6a6c-479f-86e3-b67f973cab0e" (UID: "0dafd76f-6a6c-479f-86e3-b67f973cab0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:34.146589 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.146551 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-workload-certs" pod "0dafd76f-6a6c-479f-86e3-b67f973cab0e" (UID: "0dafd76f-6a6c-479f-86e3-b67f973cab0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:34.146737 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.146649 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-envoy\") pod \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " Oct 30 20:58:34.146737 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.146691 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-token\" (UniqueName: \"kubernetes.io/projected/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-token\") pod \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " Oct 30 20:58:34.146850 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.146748 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-data" pod "0dafd76f-6a6c-479f-86e3-b67f973cab0e" (UID: "0dafd76f-6a6c-479f-86e3-b67f973cab0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:34.146850 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.146777 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-credential-socket" pod "0dafd76f-6a6c-479f-86e3-b67f973cab0e" (UID: "0dafd76f-6a6c-479f-86e3-b67f973cab0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:34.146850 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.146787 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-podinfo\") pod \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " Oct 30 20:58:34.146850 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.146828 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istiod-ca-cert\") pod \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\" (UID: \"0dafd76f-6a6c-479f-86e3-b67f973cab0e\") " Oct 30 20:58:34.149811 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.147268 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istiod-ca-cert" pod "0dafd76f-6a6c-479f-86e3-b67f973cab0e" (UID: "0dafd76f-6a6c-479f-86e3-b67f973cab0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:34.151177 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.151073 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:34.151177 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.151097 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:34.151177 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.151111 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:34.151177 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.151126 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:34.151177 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.151138 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:34.162425 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.162392 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-podinfo" pod "0dafd76f-6a6c-479f-86e3-b67f973cab0e" (UID: "0dafd76f-6a6c-479f-86e3-b67f973cab0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:34.163887 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.163853 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dafd76f-6a6c-479f-86e3-b67f973cab0e-kube-api-access-rml9s" pod "0dafd76f-6a6c-479f-86e3-b67f973cab0e" (UID: "0dafd76f-6a6c-479f-86e3-b67f973cab0e"). InnerVolumeSpecName "kube-api-access-rml9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:34.167811 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.166822 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-token" pod "0dafd76f-6a6c-479f-86e3-b67f973cab0e" (UID: "0dafd76f-6a6c-479f-86e3-b67f973cab0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:34.167811 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.166907 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-envoy" pod "0dafd76f-6a6c-479f-86e3-b67f973cab0e" (UID: "0dafd76f-6a6c-479f-86e3-b67f973cab0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:34.252752 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.252706 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-envoy\") pod \"2bdeff6d-2976-4534-8857-b04b966de5c4\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " Oct 30 20:58:34.252987 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.252838 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-workload-certs\") pod \"2bdeff6d-2976-4534-8857-b04b966de5c4\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " Oct 30 20:58:34.252987 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.252868 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-credential-socket\") pod \"2bdeff6d-2976-4534-8857-b04b966de5c4\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " Oct 30 20:58:34.252987 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.252950 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-token\" (UniqueName: \"kubernetes.io/projected/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-token\") pod \"2bdeff6d-2976-4534-8857-b04b966de5c4\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " Oct 30 20:58:34.253170 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.253005 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-podinfo\") pod \"2bdeff6d-2976-4534-8857-b04b966de5c4\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " Oct 30 20:58:34.253170 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.253098 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bdeff6d-2976-4534-8857-b04b966de5c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bdeff6d-2976-4534-8857-b04b966de5c4-istiod-ca-cert\") pod \"2bdeff6d-2976-4534-8857-b04b966de5c4\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " Oct 30 20:58:34.253170 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.253129 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-workload-socket\") pod \"2bdeff6d-2976-4534-8857-b04b966de5c4\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " Oct 30 20:58:34.253328 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.253188 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-data\") pod \"2bdeff6d-2976-4534-8857-b04b966de5c4\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " Oct 30 20:58:34.253328 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.253162 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-workload-certs" pod "2bdeff6d-2976-4534-8857-b04b966de5c4" (UID: "2bdeff6d-2976-4534-8857-b04b966de5c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:34.253328 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.253213 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-credential-socket" pod "2bdeff6d-2976-4534-8857-b04b966de5c4" (UID: "2bdeff6d-2976-4534-8857-b04b966de5c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:34.253328 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.253249 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bdeff6d-2976-4534-8857-b04b966de5c4-kube-api-access-q8xh8\" (UniqueName: \"kubernetes.io/projected/2bdeff6d-2976-4534-8857-b04b966de5c4-kube-api-access-q8xh8\") pod \"2bdeff6d-2976-4534-8857-b04b966de5c4\" (UID: \"2bdeff6d-2976-4534-8857-b04b966de5c4\") " Oct 30 20:58:34.253534 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.253404 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-workload-socket" pod "2bdeff6d-2976-4534-8857-b04b966de5c4" (UID: "2bdeff6d-2976-4534-8857-b04b966de5c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:34.253534 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.253496 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdeff6d-2976-4534-8857-b04b966de5c4-istiod-ca-cert" pod "2bdeff6d-2976-4534-8857-b04b966de5c4" (UID: "2bdeff6d-2976-4534-8857-b04b966de5c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:34.253718 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.253695 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-data" pod "2bdeff6d-2976-4534-8857-b04b966de5c4" (UID: "2bdeff6d-2976-4534-8857-b04b966de5c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:34.254354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.254025 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:34.254354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.254053 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:34.254354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.254067 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bdeff6d-2976-4534-8857-b04b966de5c4-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:34.254354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.254081 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:34.254354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.254095 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dafd76f-6a6c-479f-86e3-b67f973cab0e-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:34.254354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.254109 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:34.254354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.254123 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rml9s\" (UniqueName: \"kubernetes.io/projected/0dafd76f-6a6c-479f-86e3-b67f973cab0e-kube-api-access-rml9s\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:34.254354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.254139 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:34.254354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.254151 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:34.255753 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.255731 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-token" pod "2bdeff6d-2976-4534-8857-b04b966de5c4" (UID: "2bdeff6d-2976-4534-8857-b04b966de5c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:34.255860 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.255837 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-envoy" pod "2bdeff6d-2976-4534-8857-b04b966de5c4" (UID: "2bdeff6d-2976-4534-8857-b04b966de5c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:58:34.256015 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.255994 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdeff6d-2976-4534-8857-b04b966de5c4-kube-api-access-q8xh8" pod "2bdeff6d-2976-4534-8857-b04b966de5c4" (UID: "2bdeff6d-2976-4534-8857-b04b966de5c4"). InnerVolumeSpecName "kube-api-access-q8xh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:34.256537 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.256516 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-podinfo" pod "2bdeff6d-2976-4534-8857-b04b966de5c4" (UID: "2bdeff6d-2976-4534-8857-b04b966de5c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:58:34.359860 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.357210 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:34.359860 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.357246 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:34.359860 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.357263 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8xh8\" (UniqueName: \"kubernetes.io/projected/2bdeff6d-2976-4534-8857-b04b966de5c4-kube-api-access-q8xh8\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:34.359860 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.357278 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bdeff6d-2976-4534-8857-b04b966de5c4-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:34.520831 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.520792 4848 generic.go:356] "Generic (PLEG): container finished" podID="2bdeff6d-2976-4534-8857-b04b966de5c4" containerID="93320dffc87b4a3d3e0c23dfdc27b402462a45a1eee0a23a9aa450c8e4b64243" exitCode=0 Oct 30 20:58:34.521096 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.521081 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" Oct 30 20:58:34.521800 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.521761 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" event={"ID":"2bdeff6d-2976-4534-8857-b04b966de5c4","Type":"ContainerDied","Data":"93320dffc87b4a3d3e0c23dfdc27b402462a45a1eee0a23a9aa450c8e4b64243"} Oct 30 20:58:34.522149 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.521837 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9" event={"ID":"2bdeff6d-2976-4534-8857-b04b966de5c4","Type":"ContainerDied","Data":"2375cc8fe7da1fc283ee859cd2ff086b10143f076306a4024453ac31b62be2f0"} Oct 30 20:58:34.522149 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.521872 4848 scope.go:122] "RemoveContainer" containerID="93320dffc87b4a3d3e0c23dfdc27b402462a45a1eee0a23a9aa450c8e4b64243" Oct 30 20:58:34.529262 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.529185 4848 generic.go:356] "Generic (PLEG): container finished" podID="0dafd76f-6a6c-479f-86e3-b67f973cab0e" containerID="7c370286b08a8341526e2c1122ab46701e35259978fd2b6f7207cf8cba5cc34b" exitCode=0 Oct 30 20:58:34.529791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.529478 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" Oct 30 20:58:34.534306 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.533798 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" event={"ID":"0dafd76f-6a6c-479f-86e3-b67f973cab0e","Type":"ContainerDied","Data":"7c370286b08a8341526e2c1122ab46701e35259978fd2b6f7207cf8cba5cc34b"} Oct 30 20:58:34.534306 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.533855 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8" event={"ID":"0dafd76f-6a6c-479f-86e3-b67f973cab0e","Type":"ContainerDied","Data":"5d422990a49a6aca2a1500f2ee52bb9cae04bace2b13c0b4751a8ccca27398da"} Oct 30 20:58:34.550583 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.550472 4848 scope.go:122] "RemoveContainer" containerID="93320dffc87b4a3d3e0c23dfdc27b402462a45a1eee0a23a9aa450c8e4b64243" Oct 30 20:58:34.550908 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:58:34.550884 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93320dffc87b4a3d3e0c23dfdc27b402462a45a1eee0a23a9aa450c8e4b64243\": container with ID starting with 93320dffc87b4a3d3e0c23dfdc27b402462a45a1eee0a23a9aa450c8e4b64243 not found: ID does not exist" containerID="93320dffc87b4a3d3e0c23dfdc27b402462a45a1eee0a23a9aa450c8e4b64243" Oct 30 20:58:34.551025 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.550919 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93320dffc87b4a3d3e0c23dfdc27b402462a45a1eee0a23a9aa450c8e4b64243"} err="failed to get container status \"93320dffc87b4a3d3e0c23dfdc27b402462a45a1eee0a23a9aa450c8e4b64243\": rpc error: code = NotFound desc = could not find container \"93320dffc87b4a3d3e0c23dfdc27b402462a45a1eee0a23a9aa450c8e4b64243\": container with ID starting with 93320dffc87b4a3d3e0c23dfdc27b402462a45a1eee0a23a9aa450c8e4b64243 not found: ID does not exist" Oct 30 20:58:34.551025 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.550942 4848 scope.go:122] "RemoveContainer" containerID="7c370286b08a8341526e2c1122ab46701e35259978fd2b6f7207cf8cba5cc34b" Oct 30 20:58:34.565919 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.565397 4848 scope.go:122] "RemoveContainer" containerID="7c370286b08a8341526e2c1122ab46701e35259978fd2b6f7207cf8cba5cc34b" Oct 30 20:58:34.565919 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:58:34.565822 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c370286b08a8341526e2c1122ab46701e35259978fd2b6f7207cf8cba5cc34b\": container with ID starting with 7c370286b08a8341526e2c1122ab46701e35259978fd2b6f7207cf8cba5cc34b not found: ID does not exist" containerID="7c370286b08a8341526e2c1122ab46701e35259978fd2b6f7207cf8cba5cc34b" Oct 30 20:58:34.565919 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.565870 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c370286b08a8341526e2c1122ab46701e35259978fd2b6f7207cf8cba5cc34b"} err="failed to get container status \"7c370286b08a8341526e2c1122ab46701e35259978fd2b6f7207cf8cba5cc34b\": rpc error: code = NotFound desc = could not find container \"7c370286b08a8341526e2c1122ab46701e35259978fd2b6f7207cf8cba5cc34b\": container with ID starting with 7c370286b08a8341526e2c1122ab46701e35259978fd2b6f7207cf8cba5cc34b not found: ID does not exist" Oct 30 20:58:34.579902 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.579872 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9"] Oct 30 20:58:34.592515 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.592464 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-b46w9"] Oct 30 20:58:34.598219 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.598125 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8"] Oct 30 20:58:34.603499 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.603466 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-kn8s8"] Oct 30 20:58:34.956819 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.956783 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dafd76f-6a6c-479f-86e3-b67f973cab0e" path="/var/lib/kubelet/pods/0dafd76f-6a6c-479f-86e3-b67f973cab0e/volumes" Oct 30 20:58:34.957563 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:34.957538 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bdeff6d-2976-4534-8857-b04b966de5c4" path="/var/lib/kubelet/pods/2bdeff6d-2976-4534-8857-b04b966de5c4/volumes" Oct 30 20:58:35.884507 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:35.884476 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z"] Oct 30 20:58:35.885568 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:35.885542 4848 cpu_manager.go:414] "containerMap: removing container" podUID="2bdeff6d-2976-4534-8857-b04b966de5c4" containerName="istio-proxy" Oct 30 20:58:35.885568 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:35.885569 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bdeff6d-2976-4534-8857-b04b966de5c4" containerName="istio-proxy" Oct 30 20:58:35.885760 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:35.885667 4848 cpu_manager.go:414] "containerMap: removing container" podUID="0dafd76f-6a6c-479f-86e3-b67f973cab0e" containerName="istio-proxy" Oct 30 20:58:35.885760 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:35.885675 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dafd76f-6a6c-479f-86e3-b67f973cab0e" containerName="istio-proxy" Oct 30 20:58:35.885874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:35.885840 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dafd76f-6a6c-479f-86e3-b67f973cab0e" containerName="istio-proxy" Oct 30 20:58:35.885874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:35.885857 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bdeff6d-2976-4534-8857-b04b966de5c4" containerName="istio-proxy" Oct 30 20:58:35.891690 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:35.891670 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" Oct 30 20:58:35.906852 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:35.903141 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-70cf03-serving-cert\"" Oct 30 20:58:35.906852 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:35.906149 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z"] Oct 30 20:58:35.988462 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:35.988425 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0770330-1ef1-43c9-978c-f6d994dd9dc4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z\" (UID: \"f0770330-1ef1-43c9-978c-f6d994dd9dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" Oct 30 20:58:35.988647 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:35.988504 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0770330-1ef1-43c9-978c-f6d994dd9dc4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z\" (UID: \"f0770330-1ef1-43c9-978c-f6d994dd9dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" Oct 30 20:58:35.988647 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:35.988612 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtjg7\" (UniqueName: \"kubernetes.io/projected/f0770330-1ef1-43c9-978c-f6d994dd9dc4-kube-api-access-dtjg7\") pod \"maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z\" (UID: \"f0770330-1ef1-43c9-978c-f6d994dd9dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" Oct 30 20:58:35.988878 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:35.988847 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0770330-1ef1-43c9-978c-f6d994dd9dc4-metrics-certs\") pod \"maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z\" (UID: \"f0770330-1ef1-43c9-978c-f6d994dd9dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" Oct 30 20:58:36.098789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:36.097062 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0770330-1ef1-43c9-978c-f6d994dd9dc4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z\" (UID: \"f0770330-1ef1-43c9-978c-f6d994dd9dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" Oct 30 20:58:36.098789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:36.097145 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0770330-1ef1-43c9-978c-f6d994dd9dc4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z\" (UID: \"f0770330-1ef1-43c9-978c-f6d994dd9dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" Oct 30 20:58:36.098789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:36.097284 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtjg7\" (UniqueName: \"kubernetes.io/projected/f0770330-1ef1-43c9-978c-f6d994dd9dc4-kube-api-access-dtjg7\") pod \"maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z\" (UID: \"f0770330-1ef1-43c9-978c-f6d994dd9dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" Oct 30 20:58:36.098789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:36.097324 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0770330-1ef1-43c9-978c-f6d994dd9dc4-metrics-certs\") pod \"maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z\" (UID: \"f0770330-1ef1-43c9-978c-f6d994dd9dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" Oct 30 20:58:36.098789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:36.098599 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0770330-1ef1-43c9-978c-f6d994dd9dc4-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z\" (UID: \"f0770330-1ef1-43c9-978c-f6d994dd9dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" Oct 30 20:58:36.123393 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:36.123360 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0770330-1ef1-43c9-978c-f6d994dd9dc4-metrics-certs\") pod \"maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z\" (UID: \"f0770330-1ef1-43c9-978c-f6d994dd9dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" Oct 30 20:58:36.127672 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:36.127588 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0770330-1ef1-43c9-978c-f6d994dd9dc4-maas-api-tls\") pod \"maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z\" (UID: \"f0770330-1ef1-43c9-978c-f6d994dd9dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" Oct 30 20:58:36.147973 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:36.144560 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtjg7\" (UniqueName: \"kubernetes.io/projected/f0770330-1ef1-43c9-978c-f6d994dd9dc4-kube-api-access-dtjg7\") pod \"maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z\" (UID: \"f0770330-1ef1-43c9-978c-f6d994dd9dc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" Oct 30 20:58:36.215592 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:36.215563 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" Oct 30 20:58:36.568466 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:36.566998 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z"] Oct 30 20:58:37.582058 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.582026 4848 generic.go:356] "Generic (PLEG): container finished" podID="bed55d4a-5593-41db-84e1-fe96bb9a849b" containerID="d898fb74ee6342dd97b9e02eeb7b940d39ef5af38b64dc2b5c7fe62aaa4abdd6" exitCode=0 Oct 30 20:58:37.582531 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.582105 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" event={"ID":"bed55d4a-5593-41db-84e1-fe96bb9a849b","Type":"ContainerDied","Data":"d898fb74ee6342dd97b9e02eeb7b940d39ef5af38b64dc2b5c7fe62aaa4abdd6"} Oct 30 20:58:37.582624 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.582609 4848 scope.go:122] "RemoveContainer" containerID="d898fb74ee6342dd97b9e02eeb7b940d39ef5af38b64dc2b5c7fe62aaa4abdd6" Oct 30 20:58:37.583855 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.583809 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" event={"ID":"f0770330-1ef1-43c9-978c-f6d994dd9dc4","Type":"ContainerStarted","Data":"fc4c4b7737490bd1a6fdb03ab939a98a6d6f63c45ae99bb284b5fa10232663f2"} Oct 30 20:58:37.583967 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.583879 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" event={"ID":"f0770330-1ef1-43c9-978c-f6d994dd9dc4","Type":"ContainerStarted","Data":"1c01d9047f82def9d11634c8c11b338cd1d2c3ff548da308256ef2c3ff8620a1"} Oct 30 20:58:37.583967 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.583906 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" Oct 30 20:58:37.584445 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.584415 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:37.584500 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.584487 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:37.584541 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.584527 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:37.621411 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.621355 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" podStartSLOduration=2.142057407 podStartE2EDuration="2.621339552s" totalImagesPullingTime="479.282145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:37.62003832 +0000 UTC m=+1222.799934859" watchObservedRunningTime="2026-10-30 20:58:37.621339552 +0000 UTC m=+1222.801236058" Oct 30 20:58:37.805019 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.804981 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr"] Oct 30 20:58:37.810706 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.810676 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:37.815829 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.815444 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b46fb7-openshift-default-dockercfg-r9c4z\"" Oct 30 20:58:37.829892 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.829861 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr"] Oct 30 20:58:37.956142 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.956097 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmn42\" (UniqueName: \"kubernetes.io/projected/5c00d1f3-b911-40fd-8772-3ed4bba10d75-kube-api-access-dmn42\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:37.956538 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.956508 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-workload-certs\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:37.961810 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.961738 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-workload-socket\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:37.962114 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.962091 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istiod-ca-cert\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:37.975046 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.975000 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-data\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:37.975362 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.975313 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-token\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:37.975589 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.975569 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-podinfo\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:37.975741 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.975727 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-envoy\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:37.976004 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:37.975983 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-credential-socket\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:38.084857 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.080204 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-workload-socket\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:38.084857 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.080894 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istiod-ca-cert\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:38.084857 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.081117 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-data\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:38.084857 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.081260 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-token\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:38.084857 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.081409 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-podinfo\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:38.084857 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.081447 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-envoy\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:38.084857 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.081720 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-credential-socket\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:38.089005 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.081654 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-workload-socket\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:38.089005 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.082140 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-data\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:38.089005 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.082945 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istiod-ca-cert\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:38.089005 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.084450 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-credential-socket\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:38.098831 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.089535 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmn42\" (UniqueName: \"kubernetes.io/projected/5c00d1f3-b911-40fd-8772-3ed4bba10d75-kube-api-access-dmn42\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:38.098831 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.089684 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-workload-certs\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:38.098831 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.090095 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-workload-certs\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:38.098831 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.097959 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-token\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:38.105542 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.100827 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-podinfo\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:38.112611 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.112572 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-envoy\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:38.113509 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.113376 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmn42\" (UniqueName: \"kubernetes.io/projected/5c00d1f3-b911-40fd-8772-3ed4bba10d75-kube-api-access-dmn42\") pod \"e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:38.132746 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.132136 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:38.251981 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.246010 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx"] Oct 30 20:58:38.297006 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.265136 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.297006 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.273125 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-6c1acd-openshift-default-dockercfg-hdjqq\"" Oct 30 20:58:38.327929 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.327884 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-workload-certs\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.328207 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.328187 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-podinfo\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.328369 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.328351 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-workload-socket\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.328549 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.328531 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istiod-ca-cert\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.328678 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.328662 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-envoy\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.328895 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.328875 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhk6k\" (UniqueName: \"kubernetes.io/projected/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-kube-api-access-fhk6k\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.329056 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.329040 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-data\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.329209 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.329192 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-credential-socket\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.329433 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.329415 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-token\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.370819 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.365017 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx"] Oct 30 20:58:38.457265 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.454646 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istiod-ca-cert\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.457265 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.454707 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-envoy\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.457265 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.454755 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhk6k\" (UniqueName: \"kubernetes.io/projected/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-kube-api-access-fhk6k\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.457265 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.454832 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-data\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.457265 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.454923 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-credential-socket\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.457265 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.455035 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-token\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.457265 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.455073 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-workload-certs\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.457265 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.455121 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-podinfo\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.457265 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.455227 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-workload-socket\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.457265 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.455747 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-workload-socket\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.457265 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.456086 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-credential-socket\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.457265 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.456497 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-workload-certs\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.457265 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.456736 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istiod-ca-cert\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.457836 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.457332 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-data\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.474739 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.474642 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-envoy\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.474974 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.474711 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-podinfo\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.477946 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.477920 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhk6k\" (UniqueName: \"kubernetes.io/projected/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-kube-api-access-fhk6k\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.482021 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.481957 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-token\") pod \"e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.519666 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.519386 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" Oct 30 20:58:38.593293 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.593263 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" event={"ID":"bed55d4a-5593-41db-84e1-fe96bb9a849b","Type":"ContainerStarted","Data":"54d4855122d3d655c940739ece767ff845b185cf404e5c2c43837e255d255622"} Oct 30 20:58:38.638127 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.638091 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:38.801024 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.800993 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr"] Oct 30 20:58:38.851228 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.815006 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:38.851228 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.815117 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:38.851228 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:38.815167 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:39.044981 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.044951 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:58:39.045168 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.045086 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 20:58:39.045168 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.045140 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" podUID="bed55d4a-5593-41db-84e1-fe96bb9a849b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 20:58:39.066600 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.066568 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx"] Oct 30 20:58:39.066809 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:58:39.066758 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e08fcfc_f83f_4d7a_9ee2_cc7657244a4e.slice/crio-34f5a948c7850572920ec7690432ba3d24ca07ce4a6f179e4f6ed33885b2c975 WatchSource:0}: Error finding container 34f5a948c7850572920ec7690432ba3d24ca07ce4a6f179e4f6ed33885b2c975: Status 404 returned error can't find the container with id 34f5a948c7850572920ec7690432ba3d24ca07ce4a6f179e4f6ed33885b2c975 Oct 30 20:58:39.070871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.070828 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:39.070980 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.070920 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:39.070980 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.070964 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:39.152394 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.152358 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt"] Oct 30 20:58:39.166255 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.166231 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" Oct 30 20:58:39.188152 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.187396 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt"] Oct 30 20:58:39.233439 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.232080 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-99b00f-serving-cert\"" Oct 30 20:58:39.348838 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.348714 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrxkx\" (UniqueName: \"kubernetes.io/projected/2a94b236-339e-47ef-b5ab-49a6af417392-kube-api-access-wrxkx\") pod \"maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt\" (UID: \"2a94b236-339e-47ef-b5ab-49a6af417392\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" Oct 30 20:58:39.349014 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.348868 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a94b236-339e-47ef-b5ab-49a6af417392-metrics-certs\") pod \"maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt\" (UID: \"2a94b236-339e-47ef-b5ab-49a6af417392\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" Oct 30 20:58:39.349014 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.348904 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a94b236-339e-47ef-b5ab-49a6af417392-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt\" (UID: \"2a94b236-339e-47ef-b5ab-49a6af417392\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" Oct 30 20:58:39.349014 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.348974 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a94b236-339e-47ef-b5ab-49a6af417392-maas-api-tls\") pod \"maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt\" (UID: \"2a94b236-339e-47ef-b5ab-49a6af417392\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" Oct 30 20:58:39.473383 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.460527 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a94b236-339e-47ef-b5ab-49a6af417392-metrics-certs\") pod \"maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt\" (UID: \"2a94b236-339e-47ef-b5ab-49a6af417392\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" Oct 30 20:58:39.473383 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.470877 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a94b236-339e-47ef-b5ab-49a6af417392-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt\" (UID: \"2a94b236-339e-47ef-b5ab-49a6af417392\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" Oct 30 20:58:39.473383 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.470988 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a94b236-339e-47ef-b5ab-49a6af417392-maas-api-tls\") pod \"maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt\" (UID: \"2a94b236-339e-47ef-b5ab-49a6af417392\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" Oct 30 20:58:39.473383 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.471873 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a94b236-339e-47ef-b5ab-49a6af417392-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt\" (UID: \"2a94b236-339e-47ef-b5ab-49a6af417392\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" Oct 30 20:58:39.473383 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.472146 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrxkx\" (UniqueName: \"kubernetes.io/projected/2a94b236-339e-47ef-b5ab-49a6af417392-kube-api-access-wrxkx\") pod \"maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt\" (UID: \"2a94b236-339e-47ef-b5ab-49a6af417392\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" Oct 30 20:58:39.474315 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.474281 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a94b236-339e-47ef-b5ab-49a6af417392-metrics-certs\") pod \"maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt\" (UID: \"2a94b236-339e-47ef-b5ab-49a6af417392\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" Oct 30 20:58:39.475987 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.475966 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a94b236-339e-47ef-b5ab-49a6af417392-maas-api-tls\") pod \"maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt\" (UID: \"2a94b236-339e-47ef-b5ab-49a6af417392\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" Oct 30 20:58:39.484246 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.484223 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrxkx\" (UniqueName: \"kubernetes.io/projected/2a94b236-339e-47ef-b5ab-49a6af417392-kube-api-access-wrxkx\") pod \"maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt\" (UID: \"2a94b236-339e-47ef-b5ab-49a6af417392\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" Oct 30 20:58:39.500980 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.500963 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" Oct 30 20:58:39.620996 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.619180 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" event={"ID":"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e","Type":"ContainerStarted","Data":"2d358be9ef0986f7d0e7776ea57191f5e2a4aa52162894a3138941fb7128cceb"} Oct 30 20:58:39.620996 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.619252 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" event={"ID":"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e","Type":"ContainerStarted","Data":"34f5a948c7850572920ec7690432ba3d24ca07ce4a6f179e4f6ed33885b2c975"} Oct 30 20:58:39.631123 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.624437 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" event={"ID":"5c00d1f3-b911-40fd-8772-3ed4bba10d75","Type":"ContainerStarted","Data":"26f56a772ac1cb31314790b81e470d8800e6e09c9790f2c14135946a6b168088"} Oct 30 20:58:39.631123 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.624490 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" event={"ID":"5c00d1f3-b911-40fd-8772-3ed4bba10d75","Type":"ContainerStarted","Data":"7bdc3cb0854021e5f8e7e0d94ee843dc1f0dc2f1c068e2ffcc367de6767d0360"} Oct 30 20:58:39.652054 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.638911 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:39.652611 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.652566 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" podStartSLOduration=1.6525500370000001 podStartE2EDuration="1.652550037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:39.648315083 +0000 UTC m=+1224.828211602" watchObservedRunningTime="2026-10-30 20:58:39.652550037 +0000 UTC m=+1224.832446540" Oct 30 20:58:39.670799 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:39.668130 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" podStartSLOduration=2.66811366 podStartE2EDuration="2.66811366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:39.667485757 +0000 UTC m=+1224.847382265" watchObservedRunningTime="2026-10-30 20:58:39.66811366 +0000 UTC m=+1224.848010167" Oct 30 20:58:40.000796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:40.000747 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt"] Oct 30 20:58:40.002332 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:58:40.002302 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a94b236_339e_47ef_b5ab_49a6af417392.slice/crio-5812b8bf5747c18c331b36aeb1f4c52c0b41c9fe1b44f16160e0914c66d981cb WatchSource:0}: Error finding container 5812b8bf5747c18c331b36aeb1f4c52c0b41c9fe1b44f16160e0914c66d981cb: Status 404 returned error can't find the container with id 5812b8bf5747c18c331b36aeb1f4c52c0b41c9fe1b44f16160e0914c66d981cb Oct 30 20:58:40.045147 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:40.045112 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 20:58:40.045504 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:40.045446 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" podUID="bed55d4a-5593-41db-84e1-fe96bb9a849b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 20:58:40.133802 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:40.133136 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:40.140825 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:40.140758 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:40.475844 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:40.475819 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r" Oct 30 20:58:40.476037 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:40.475884 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr" Oct 30 20:58:40.650875 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:40.650847 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:40.658801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:40.655882 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" event={"ID":"2a94b236-339e-47ef-b5ab-49a6af417392","Type":"ContainerStarted","Data":"5812b8bf5747c18c331b36aeb1f4c52c0b41c9fe1b44f16160e0914c66d981cb"} Oct 30 20:58:40.658801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:40.656347 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:40.658801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:40.656508 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:40.658801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:40.657358 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:58:40.658801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:40.657485 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:58:41.044431 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:41.044395 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 20:58:41.044612 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:41.044447 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" podUID="bed55d4a-5593-41db-84e1-fe96bb9a849b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 20:58:41.664521 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:41.664478 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" event={"ID":"2a94b236-339e-47ef-b5ab-49a6af417392","Type":"ContainerStarted","Data":"5285e7ca950944fa7788929da68d3c39bcd9367076b7054c98177f743bcc6f18"} Oct 30 20:58:41.665042 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:41.664735 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" Oct 30 20:58:41.682611 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:41.682566 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" podStartSLOduration=2.124094593 podStartE2EDuration="2.682553558s" totalImagesPullingTime="558.458965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:41.681134061 +0000 UTC m=+1226.861030569" watchObservedRunningTime="2026-10-30 20:58:41.682553558 +0000 UTC m=+1226.862450114" Oct 30 20:58:42.044354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:42.044308 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 20:58:42.044523 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:42.044396 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" podUID="bed55d4a-5593-41db-84e1-fe96bb9a849b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 20:58:42.531176 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:42.531139 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw" Oct 30 20:58:42.532653 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:42.532630 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt" Oct 30 20:58:43.044532 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.044491 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 20:58:43.044949 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.044549 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" podUID="bed55d4a-5593-41db-84e1-fe96bb9a849b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 20:58:43.339567 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.339490 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv"] Oct 30 20:58:43.344205 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.344178 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv" Oct 30 20:58:43.353154 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.353129 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-6c1acd-dockercfg-mgdz6\"" Oct 30 20:58:43.353484 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.353465 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-6c1acd\"" Oct 30 20:58:43.354779 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.354742 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv"] Oct 30 20:58:43.378279 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.378249 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9"] Oct 30 20:58:43.382673 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.382650 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9" Oct 30 20:58:43.390904 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.390873 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9"] Oct 30 20:58:43.444522 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.444403 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hlw9\" (UniqueName: \"kubernetes.io/projected/8867aa05-25c2-49c6-bd30-0f8cc53c985e-kube-api-access-6hlw9\") pod \"payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv\" (UID: \"8867aa05-25c2-49c6-bd30-0f8cc53c985e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv" Oct 30 20:58:43.444780 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.444722 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8867aa05-25c2-49c6-bd30-0f8cc53c985e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv\" (UID: \"8867aa05-25c2-49c6-bd30-0f8cc53c985e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv" Oct 30 20:58:43.572792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.571803 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8867aa05-25c2-49c6-bd30-0f8cc53c985e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv\" (UID: \"8867aa05-25c2-49c6-bd30-0f8cc53c985e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv" Oct 30 20:58:43.572792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.571938 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hlw9\" (UniqueName: \"kubernetes.io/projected/8867aa05-25c2-49c6-bd30-0f8cc53c985e-kube-api-access-6hlw9\") pod \"payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv\" (UID: \"8867aa05-25c2-49c6-bd30-0f8cc53c985e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv" Oct 30 20:58:43.576785 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.573654 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8867aa05-25c2-49c6-bd30-0f8cc53c985e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv\" (UID: \"8867aa05-25c2-49c6-bd30-0f8cc53c985e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv" Oct 30 20:58:43.686341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.681003 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8588f639-d98d-48f8-8d35-fdb56ad83f6c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9\" (UID: \"8588f639-d98d-48f8-8d35-fdb56ad83f6c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9" Oct 30 20:58:43.686341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.681303 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx4g6\" (UniqueName: \"kubernetes.io/projected/8588f639-d98d-48f8-8d35-fdb56ad83f6c-kube-api-access-rx4g6\") pod \"payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9\" (UID: \"8588f639-d98d-48f8-8d35-fdb56ad83f6c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9" Oct 30 20:58:43.735961 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.735475 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hlw9\" (UniqueName: \"kubernetes.io/projected/8867aa05-25c2-49c6-bd30-0f8cc53c985e-kube-api-access-6hlw9\") pod \"payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv\" (UID: \"8867aa05-25c2-49c6-bd30-0f8cc53c985e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv" Oct 30 20:58:43.749244 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.749177 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" Oct 30 20:58:43.783359 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.783315 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx4g6\" (UniqueName: \"kubernetes.io/projected/8588f639-d98d-48f8-8d35-fdb56ad83f6c-kube-api-access-rx4g6\") pod \"payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9\" (UID: \"8588f639-d98d-48f8-8d35-fdb56ad83f6c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9" Oct 30 20:58:43.783555 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.783439 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8588f639-d98d-48f8-8d35-fdb56ad83f6c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9\" (UID: \"8588f639-d98d-48f8-8d35-fdb56ad83f6c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9" Oct 30 20:58:43.784674 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.784649 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8588f639-d98d-48f8-8d35-fdb56ad83f6c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9\" (UID: \"8588f639-d98d-48f8-8d35-fdb56ad83f6c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9" Oct 30 20:58:43.823801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.821496 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx4g6\" (UniqueName: \"kubernetes.io/projected/8588f639-d98d-48f8-8d35-fdb56ad83f6c-kube-api-access-rx4g6\") pod \"payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9\" (UID: \"8588f639-d98d-48f8-8d35-fdb56ad83f6c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9" Oct 30 20:58:43.967157 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:43.967005 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv" Oct 30 20:58:44.064269 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.064237 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 20:58:44.064923 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.064888 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" podUID="bed55d4a-5593-41db-84e1-fe96bb9a849b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 20:58:44.065579 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.065547 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9" Oct 30 20:58:44.174869 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.174833 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:58:44.435289 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.435241 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv"] Oct 30 20:58:44.493817 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.493783 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9"] Oct 30 20:58:44.727080 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.727047 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9" event={"ID":"8588f639-d98d-48f8-8d35-fdb56ad83f6c","Type":"ContainerStarted","Data":"dea9d021d2031e0cc50da912ff92012d0fa5326a88dfed1fec63acd1b8fc5716"} Oct 30 20:58:44.727239 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.727124 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9" event={"ID":"8588f639-d98d-48f8-8d35-fdb56ad83f6c","Type":"ContainerStarted","Data":"5715e58465aef5a3eaa5f2aa3b2dfa354b3c2a831c59a2d32f4efcb71779de67"} Oct 30 20:58:44.727239 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.727160 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9" Oct 30 20:58:44.729297 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.729269 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv" event={"ID":"8867aa05-25c2-49c6-bd30-0f8cc53c985e","Type":"ContainerStarted","Data":"2f26ed374f182c03dc5736d30202c6b7b9035c72162b28ca967c0583999b5f2d"} Oct 30 20:58:44.729424 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.729319 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv" Oct 30 20:58:44.729424 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.729335 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv" event={"ID":"8867aa05-25c2-49c6-bd30-0f8cc53c985e","Type":"ContainerStarted","Data":"7d77facb4b89f46a1ffdeb6301f5c9042761060484f9900f35edc449a3953717"} Oct 30 20:58:44.752029 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.751940 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9" podStartSLOduration=1.751923606 podStartE2EDuration="1.751923606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:44.746680146 +0000 UTC m=+1229.926576653" watchObservedRunningTime="2026-10-30 20:58:44.751923606 +0000 UTC m=+1229.931820114" Oct 30 20:58:44.758492 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.758464 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng"] Oct 30 20:58:44.764198 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.764176 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng" Oct 30 20:58:44.764718 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.764680 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv" podStartSLOduration=1.764665846 podStartE2EDuration="1.764665846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:44.763018134 +0000 UTC m=+1229.942914643" watchObservedRunningTime="2026-10-30 20:58:44.764665846 +0000 UTC m=+1229.944562395" Oct 30 20:58:44.772197 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.772171 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng"] Oct 30 20:58:44.772298 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.772176 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b46fb7\"" Oct 30 20:58:44.823602 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.815819 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f6682a-f8be-4abf-99e7-0977b3e61600-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng\" (UID: \"c7f6682a-f8be-4abf-99e7-0977b3e61600\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng" Oct 30 20:58:44.823602 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.817085 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkxz9\" (UniqueName: \"kubernetes.io/projected/c7f6682a-f8be-4abf-99e7-0977b3e61600-kube-api-access-vkxz9\") pod \"payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng\" (UID: \"c7f6682a-f8be-4abf-99e7-0977b3e61600\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng" Oct 30 20:58:44.922820 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.918868 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f6682a-f8be-4abf-99e7-0977b3e61600-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng\" (UID: \"c7f6682a-f8be-4abf-99e7-0977b3e61600\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng" Oct 30 20:58:44.922820 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.919056 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkxz9\" (UniqueName: \"kubernetes.io/projected/c7f6682a-f8be-4abf-99e7-0977b3e61600-kube-api-access-vkxz9\") pod \"payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng\" (UID: \"c7f6682a-f8be-4abf-99e7-0977b3e61600\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng" Oct 30 20:58:44.922820 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.920708 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f6682a-f8be-4abf-99e7-0977b3e61600-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng\" (UID: \"c7f6682a-f8be-4abf-99e7-0977b3e61600\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng" Oct 30 20:58:45.013843 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:44.983698 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkxz9\" (UniqueName: \"kubernetes.io/projected/c7f6682a-f8be-4abf-99e7-0977b3e61600-kube-api-access-vkxz9\") pod \"payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng\" (UID: \"c7f6682a-f8be-4abf-99e7-0977b3e61600\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng" Oct 30 20:58:45.059827 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:45.059503 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:58:45.066791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:45.066732 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:58:45.082018 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:45.081986 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng" Oct 30 20:58:45.382415 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:45.382384 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng"] Oct 30 20:58:45.383429 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:58:45.383404 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f6682a_f8be_4abf_99e7_0977b3e61600.slice/crio-2b18076e8628f0aeb7ab1b9f60bcd3603dff3dfea4a3ba4060ce8c4abc35ec2e WatchSource:0}: Error finding container 2b18076e8628f0aeb7ab1b9f60bcd3603dff3dfea4a3ba4060ce8c4abc35ec2e: Status 404 returned error can't find the container with id 2b18076e8628f0aeb7ab1b9f60bcd3603dff3dfea4a3ba4060ce8c4abc35ec2e Oct 30 20:58:45.746973 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:45.746922 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng" event={"ID":"c7f6682a-f8be-4abf-99e7-0977b3e61600","Type":"ContainerStarted","Data":"025b78c62bbc38e363008902b99ce8530ddb409ad8d5dbcd10c1422ad4cba4e3"} Oct 30 20:58:45.747162 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:45.746986 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng" event={"ID":"c7f6682a-f8be-4abf-99e7-0977b3e61600","Type":"ContainerStarted","Data":"2b18076e8628f0aeb7ab1b9f60bcd3603dff3dfea4a3ba4060ce8c4abc35ec2e"} Oct 30 20:58:45.747162 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:45.747036 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng" Oct 30 20:58:45.767378 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:45.767324 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng" podStartSLOduration=1.7673085689999999 podStartE2EDuration="1.767308569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:45.765790181 +0000 UTC m=+1230.945686683" watchObservedRunningTime="2026-10-30 20:58:45.767308569 +0000 UTC m=+1230.947205076" Oct 30 20:58:47.678904 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:47.678873 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" Oct 30 20:58:49.368045 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:49.368006 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx"] Oct 30 20:58:49.378234 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:49.378207 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd"] Oct 30 20:58:52.585790 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:58:52.583423 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:58:54.369788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:54.368836 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx" podUID="598f3e3d-8117-4d80-b192-8fa7799a2420" containerName="payload-processing" containerID="cri-o://4beeaddbb23e1b435fd8cf0724cd71de506ef7265d5680fe8e8d0382a74867d1" gracePeriod=25 Oct 30 20:58:54.379399 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:54.378837 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd" podUID="5d07b7cf-5e5f-470d-82d8-6ff84370fdf3" containerName="payload-pre-processing" containerID="cri-o://ce19c38f0aabb2966bfb3c1379f5289c2d31a6485368318bf49a2b53b59f964a" gracePeriod=25 Oct 30 20:58:54.903200 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:54.902966 4848 generic.go:356] "Generic (PLEG): container finished" podID="5d07b7cf-5e5f-470d-82d8-6ff84370fdf3" containerID="ce19c38f0aabb2966bfb3c1379f5289c2d31a6485368318bf49a2b53b59f964a" exitCode=0 Oct 30 20:58:54.903200 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:54.903132 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd" event={"ID":"5d07b7cf-5e5f-470d-82d8-6ff84370fdf3","Type":"ContainerDied","Data":"ce19c38f0aabb2966bfb3c1379f5289c2d31a6485368318bf49a2b53b59f964a"} Oct 30 20:58:54.905316 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:54.904905 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx" Oct 30 20:58:54.905912 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:54.905654 4848 generic.go:356] "Generic (PLEG): container finished" podID="598f3e3d-8117-4d80-b192-8fa7799a2420" containerID="4beeaddbb23e1b435fd8cf0724cd71de506ef7265d5680fe8e8d0382a74867d1" exitCode=0 Oct 30 20:58:54.905912 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:54.905692 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx" event={"ID":"598f3e3d-8117-4d80-b192-8fa7799a2420","Type":"ContainerDied","Data":"4beeaddbb23e1b435fd8cf0724cd71de506ef7265d5680fe8e8d0382a74867d1"} Oct 30 20:58:54.905912 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:54.905736 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx" event={"ID":"598f3e3d-8117-4d80-b192-8fa7799a2420","Type":"ContainerDied","Data":"d7b8929ba8540c0bd28626da9b64ebd9fa48f19df6f031ac8c18e466af9f6a89"} Oct 30 20:58:54.905912 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:54.905784 4848 scope.go:122] "RemoveContainer" containerID="4beeaddbb23e1b435fd8cf0724cd71de506ef7265d5680fe8e8d0382a74867d1" Oct 30 20:58:54.972815 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:54.972444 4848 scope.go:122] "RemoveContainer" containerID="4beeaddbb23e1b435fd8cf0724cd71de506ef7265d5680fe8e8d0382a74867d1" Oct 30 20:58:54.972815 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:58:54.972796 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4beeaddbb23e1b435fd8cf0724cd71de506ef7265d5680fe8e8d0382a74867d1\": container with ID starting with 4beeaddbb23e1b435fd8cf0724cd71de506ef7265d5680fe8e8d0382a74867d1 not found: ID does not exist" containerID="4beeaddbb23e1b435fd8cf0724cd71de506ef7265d5680fe8e8d0382a74867d1" Oct 30 20:58:54.973008 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:54.972828 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4beeaddbb23e1b435fd8cf0724cd71de506ef7265d5680fe8e8d0382a74867d1"} err="failed to get container status \"4beeaddbb23e1b435fd8cf0724cd71de506ef7265d5680fe8e8d0382a74867d1\": rpc error: code = NotFound desc = could not find container \"4beeaddbb23e1b435fd8cf0724cd71de506ef7265d5680fe8e8d0382a74867d1\": container with ID starting with 4beeaddbb23e1b435fd8cf0724cd71de506ef7265d5680fe8e8d0382a74867d1 not found: ID does not exist" Oct 30 20:58:54.996637 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:54.996610 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd" Oct 30 20:58:54.998524 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:54.998500 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d07b7cf-5e5f-470d-82d8-6ff84370fdf3-kube-api-access-lk8v2\" (UniqueName: \"kubernetes.io/projected/5d07b7cf-5e5f-470d-82d8-6ff84370fdf3-kube-api-access-lk8v2\") pod \"5d07b7cf-5e5f-470d-82d8-6ff84370fdf3\" (UID: \"5d07b7cf-5e5f-470d-82d8-6ff84370fdf3\") " Oct 30 20:58:54.998615 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:54.998534 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d07b7cf-5e5f-470d-82d8-6ff84370fdf3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d07b7cf-5e5f-470d-82d8-6ff84370fdf3-plugins-config-volume\") pod \"5d07b7cf-5e5f-470d-82d8-6ff84370fdf3\" (UID: \"5d07b7cf-5e5f-470d-82d8-6ff84370fdf3\") " Oct 30 20:58:54.998737 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:54.998717 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/598f3e3d-8117-4d80-b192-8fa7799a2420-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/598f3e3d-8117-4d80-b192-8fa7799a2420-plugins-config-volume\") pod \"598f3e3d-8117-4d80-b192-8fa7799a2420\" (UID: \"598f3e3d-8117-4d80-b192-8fa7799a2420\") " Oct 30 20:58:54.998909 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:54.998892 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/598f3e3d-8117-4d80-b192-8fa7799a2420-kube-api-access-22g25\" (UniqueName: \"kubernetes.io/projected/598f3e3d-8117-4d80-b192-8fa7799a2420-kube-api-access-22g25\") pod \"598f3e3d-8117-4d80-b192-8fa7799a2420\" (UID: \"598f3e3d-8117-4d80-b192-8fa7799a2420\") " Oct 30 20:58:54.999056 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:54.999032 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d07b7cf-5e5f-470d-82d8-6ff84370fdf3-plugins-config-volume" pod "5d07b7cf-5e5f-470d-82d8-6ff84370fdf3" (UID: "5d07b7cf-5e5f-470d-82d8-6ff84370fdf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:54.999136 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:54.999111 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598f3e3d-8117-4d80-b192-8fa7799a2420-plugins-config-volume" pod "598f3e3d-8117-4d80-b192-8fa7799a2420" (UID: "598f3e3d-8117-4d80-b192-8fa7799a2420"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:58:54.999371 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:54.999352 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/598f3e3d-8117-4d80-b192-8fa7799a2420-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:54.999428 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:54.999377 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d07b7cf-5e5f-470d-82d8-6ff84370fdf3-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:55.001068 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.001039 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598f3e3d-8117-4d80-b192-8fa7799a2420-kube-api-access-22g25" pod "598f3e3d-8117-4d80-b192-8fa7799a2420" (UID: "598f3e3d-8117-4d80-b192-8fa7799a2420"). InnerVolumeSpecName "kube-api-access-22g25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:55.001332 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.001310 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d07b7cf-5e5f-470d-82d8-6ff84370fdf3-kube-api-access-lk8v2" pod "5d07b7cf-5e5f-470d-82d8-6ff84370fdf3" (UID: "5d07b7cf-5e5f-470d-82d8-6ff84370fdf3"). InnerVolumeSpecName "kube-api-access-lk8v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:58:55.100406 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.100371 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22g25\" (UniqueName: \"kubernetes.io/projected/598f3e3d-8117-4d80-b192-8fa7799a2420-kube-api-access-22g25\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:55.100406 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.100404 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk8v2\" (UniqueName: \"kubernetes.io/projected/5d07b7cf-5e5f-470d-82d8-6ff84370fdf3-kube-api-access-lk8v2\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:58:55.695762 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.695724 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d"] Oct 30 20:58:55.696471 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.696455 4848 cpu_manager.go:414] "containerMap: removing container" podUID="598f3e3d-8117-4d80-b192-8fa7799a2420" containerName="payload-processing" Oct 30 20:58:55.696471 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.696472 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="598f3e3d-8117-4d80-b192-8fa7799a2420" containerName="payload-processing" Oct 30 20:58:55.696559 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.696485 4848 cpu_manager.go:414] "containerMap: removing container" podUID="5d07b7cf-5e5f-470d-82d8-6ff84370fdf3" containerName="payload-pre-processing" Oct 30 20:58:55.696559 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.696496 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d07b7cf-5e5f-470d-82d8-6ff84370fdf3" containerName="payload-pre-processing" Oct 30 20:58:55.696689 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.696678 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="598f3e3d-8117-4d80-b192-8fa7799a2420" containerName="payload-processing" Oct 30 20:58:55.696732 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.696691 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d07b7cf-5e5f-470d-82d8-6ff84370fdf3" containerName="payload-pre-processing" Oct 30 20:58:55.700504 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.700490 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" Oct 30 20:58:55.720803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.709082 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-6c1acd-serving-cert\"" Oct 30 20:58:55.720803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.715055 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d"] Oct 30 20:58:55.737229 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.737199 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/260ef59c-c837-48ca-941b-f9304c1c5d96-maas-api-tls\") pod \"maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d\" (UID: \"260ef59c-c837-48ca-941b-f9304c1c5d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" Oct 30 20:58:55.737566 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.737548 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/260ef59c-c837-48ca-941b-f9304c1c5d96-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d\" (UID: \"260ef59c-c837-48ca-941b-f9304c1c5d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" Oct 30 20:58:55.739204 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.739182 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260ef59c-c837-48ca-941b-f9304c1c5d96-metrics-certs\") pod \"maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d\" (UID: \"260ef59c-c837-48ca-941b-f9304c1c5d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" Oct 30 20:58:55.739420 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.739400 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqtv5\" (UniqueName: \"kubernetes.io/projected/260ef59c-c837-48ca-941b-f9304c1c5d96-kube-api-access-xqtv5\") pod \"maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d\" (UID: \"260ef59c-c837-48ca-941b-f9304c1c5d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" Oct 30 20:58:55.750964 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.750924 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9" Oct 30 20:58:55.752748 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.752381 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv" Oct 30 20:58:55.843035 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.840982 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/260ef59c-c837-48ca-941b-f9304c1c5d96-maas-api-tls\") pod \"maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d\" (UID: \"260ef59c-c837-48ca-941b-f9304c1c5d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" Oct 30 20:58:55.843035 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.841108 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/260ef59c-c837-48ca-941b-f9304c1c5d96-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d\" (UID: \"260ef59c-c837-48ca-941b-f9304c1c5d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" Oct 30 20:58:55.843035 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.841145 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260ef59c-c837-48ca-941b-f9304c1c5d96-metrics-certs\") pod \"maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d\" (UID: \"260ef59c-c837-48ca-941b-f9304c1c5d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" Oct 30 20:58:55.843035 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.841192 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqtv5\" (UniqueName: \"kubernetes.io/projected/260ef59c-c837-48ca-941b-f9304c1c5d96-kube-api-access-xqtv5\") pod \"maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d\" (UID: \"260ef59c-c837-48ca-941b-f9304c1c5d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" Oct 30 20:58:55.843035 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.842717 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/260ef59c-c837-48ca-941b-f9304c1c5d96-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d\" (UID: \"260ef59c-c837-48ca-941b-f9304c1c5d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" Oct 30 20:58:55.845879 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.845842 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260ef59c-c837-48ca-941b-f9304c1c5d96-metrics-certs\") pod \"maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d\" (UID: \"260ef59c-c837-48ca-941b-f9304c1c5d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" Oct 30 20:58:55.850233 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.850212 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/260ef59c-c837-48ca-941b-f9304c1c5d96-maas-api-tls\") pod \"maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d\" (UID: \"260ef59c-c837-48ca-941b-f9304c1c5d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" Oct 30 20:58:55.850434 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:55.850409 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqtv5\" (UniqueName: \"kubernetes.io/projected/260ef59c-c837-48ca-941b-f9304c1c5d96-kube-api-access-xqtv5\") pod \"maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d\" (UID: \"260ef59c-c837-48ca-941b-f9304c1c5d96\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" Oct 30 20:58:56.017680 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:56.017475 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd" event={"ID":"5d07b7cf-5e5f-470d-82d8-6ff84370fdf3","Type":"ContainerDied","Data":"dc190bfbe738417d81dcfad9657703b5fe5eb4b58c382f7a6419b7059c7464a9"} Oct 30 20:58:56.017680 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:56.017551 4848 scope.go:122] "RemoveContainer" containerID="ce19c38f0aabb2966bfb3c1379f5289c2d31a6485368318bf49a2b53b59f964a" Oct 30 20:58:56.017928 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:56.017704 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd" Oct 30 20:58:56.040803 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:56.037680 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx" Oct 30 20:58:56.042425 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:56.042401 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" Oct 30 20:58:56.074780 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:56.074735 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd"] Oct 30 20:58:56.082019 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:56.081986 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-955eb7-5bd564fc7f-4hnrd"] Oct 30 20:58:56.101067 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:56.101030 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx"] Oct 30 20:58:56.126790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:56.120705 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-955eb7-df97b6dc9-bn4cx"] Oct 30 20:58:56.546110 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:56.546077 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d"] Oct 30 20:58:56.757695 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:56.757647 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng" Oct 30 20:58:56.952054 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:56.952013 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="598f3e3d-8117-4d80-b192-8fa7799a2420" path="/var/lib/kubelet/pods/598f3e3d-8117-4d80-b192-8fa7799a2420/volumes" Oct 30 20:58:56.952486 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:56.952468 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d07b7cf-5e5f-470d-82d8-6ff84370fdf3" path="/var/lib/kubelet/pods/5d07b7cf-5e5f-470d-82d8-6ff84370fdf3/volumes" Oct 30 20:58:57.169937 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:57.169886 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" event={"ID":"260ef59c-c837-48ca-941b-f9304c1c5d96","Type":"ContainerStarted","Data":"19ac29bfaf731032867bc644813a2be76d6bbbd0817101a213b0588014d409bf"} Oct 30 20:58:58.187525 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:58.187485 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" event={"ID":"260ef59c-c837-48ca-941b-f9304c1c5d96","Type":"ContainerStarted","Data":"62e665b6d61305011f39f150a50150b40e62b3c7a34709424d8ac7edbc168e31"} Oct 30 20:58:58.188023 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:58.187697 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" Oct 30 20:58:58.189335 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:58.189313 4848 generic.go:356] "Generic (PLEG): container finished" podID="8fef8496-2113-44ff-9694-038139e4b194" containerID="fd6a60993c23f9881c200d1afe9b1f5c92930b5fb5b92a8cd853a88518ef0d85" exitCode=0 Oct 30 20:58:58.189439 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:58.189393 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" event={"ID":"8fef8496-2113-44ff-9694-038139e4b194","Type":"ContainerDied","Data":"fd6a60993c23f9881c200d1afe9b1f5c92930b5fb5b92a8cd853a88518ef0d85"} Oct 30 20:58:58.189864 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:58.189847 4848 scope.go:122] "RemoveContainer" containerID="fd6a60993c23f9881c200d1afe9b1f5c92930b5fb5b92a8cd853a88518ef0d85" Oct 30 20:58:58.191435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:58.191408 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:58.191503 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:58.191474 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:58.191544 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:58.191509 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:58:58.209481 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:58.209438 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" podStartSLOduration=2.629751236 podStartE2EDuration="3.209426423s" totalImagesPullingTime="579.675187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:58:58.205601616 +0000 UTC m=+1243.385498341" watchObservedRunningTime="2026-10-30 20:58:58.209426423 +0000 UTC m=+1243.389322929" Oct 30 20:58:59.200657 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:59.200621 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" event={"ID":"8fef8496-2113-44ff-9694-038139e4b194","Type":"ContainerStarted","Data":"40686b69066115ffd9c0019177fdfc0cb5654d8944bd3b7657fba97a5c2d31e5"} Oct 30 20:58:59.954053 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:58:59.953948 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:59:00.068793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:00.067183 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 20:59:00.068793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:00.067262 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" podUID="8fef8496-2113-44ff-9694-038139e4b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 20:59:00.945338 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:00.945304 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 20:59:00.945752 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:00.945363 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" podUID="8fef8496-2113-44ff-9694-038139e4b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 20:59:01.945318 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:01.945278 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 20:59:01.945583 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:01.945334 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" podUID="8fef8496-2113-44ff-9694-038139e4b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 20:59:02.200346 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:59:02.200258 4848 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eace1e0_9882_47ec_8c5d_927d8de6ffe6.slice/crio-conmon-63ea6f5c00754eacadaeefa89d30b59385ea9c1909abfcbbb1d240648729a9ec.scope\": RecentStats: unable to find data in memory cache]" Oct 30 20:59:02.257579 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:02.257544 4848 generic.go:356] "Generic (PLEG): container finished" podID="2eace1e0-9882-47ec-8c5d-927d8de6ffe6" containerID="63ea6f5c00754eacadaeefa89d30b59385ea9c1909abfcbbb1d240648729a9ec" exitCode=0 Oct 30 20:59:02.257757 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:02.257616 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" event={"ID":"2eace1e0-9882-47ec-8c5d-927d8de6ffe6","Type":"ContainerDied","Data":"63ea6f5c00754eacadaeefa89d30b59385ea9c1909abfcbbb1d240648729a9ec"} Oct 30 20:59:02.258159 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:02.258145 4848 scope.go:122] "RemoveContainer" containerID="63ea6f5c00754eacadaeefa89d30b59385ea9c1909abfcbbb1d240648729a9ec" Oct 30 20:59:02.259861 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:02.259834 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:02.259931 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:02.259904 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:02.259969 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:02.259938 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:02.945700 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:02.945665 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 20:59:02.946121 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:02.945715 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" podUID="8fef8496-2113-44ff-9694-038139e4b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 20:59:03.290848 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:03.290675 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" event={"ID":"2eace1e0-9882-47ec-8c5d-927d8de6ffe6","Type":"ContainerStarted","Data":"7546b89e4af05979052c575f4e4dd98bc22e433460a3c9c3b2c5a8b8d592502a"} Oct 30 20:59:03.689689 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:03.689565 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:59:03.690173 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:03.690110 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 20:59:03.690408 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:03.690363 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" podUID="2eace1e0-9882-47ec-8c5d-927d8de6ffe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 20:59:03.946860 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:03.946213 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 20:59:03.946860 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:03.946275 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" podUID="8fef8496-2113-44ff-9694-038139e4b194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 20:59:03.955971 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:03.955940 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:59:04.298585 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:04.298499 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" Oct 30 20:59:04.688492 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:04.688458 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 20:59:04.688686 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:04.688532 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" podUID="2eace1e0-9882-47ec-8c5d-927d8de6ffe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 20:59:05.006198 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:05.006114 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:59:05.029865 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:05.029010 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 20:59:05.690788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:05.689322 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 20:59:05.690788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:05.689400 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" podUID="2eace1e0-9882-47ec-8c5d-927d8de6ffe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 20:59:06.688092 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:06.688059 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 20:59:06.688522 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:06.688112 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" podUID="2eace1e0-9882-47ec-8c5d-927d8de6ffe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 20:59:07.688812 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:07.688756 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 20:59:07.689207 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:07.688830 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" podUID="2eace1e0-9882-47ec-8c5d-927d8de6ffe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 20:59:08.688070 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:08.688037 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 20:59:08.688246 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:08.688100 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" podUID="2eace1e0-9882-47ec-8c5d-927d8de6ffe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 20:59:09.046536 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:09.044213 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:59:09.783796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:09.783125 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:59:09.798874 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:09.798039 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 20:59:11.142799 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:11.142745 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz"] Oct 30 20:59:11.170846 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:11.163191 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8"] Oct 30 20:59:12.100793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:12.097222 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h"] Oct 30 20:59:12.124793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:12.113527 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b"] Oct 30 20:59:12.237790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:12.237400 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:12.276035 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:12.275930 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded" Oct 30 20:59:12.356662 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:12.354860 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:12.569189 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:12.569146 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Oct 30 20:59:12.704231 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:12.704179 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 20:59:13.020292 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:13.020172 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 20:59:13.242110 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:13.242065 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:13.243201 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:13.243175 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:59:13.361968 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:13.361868 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 20:59:13.362755 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:13.362731 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:59:13.574372 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:13.573335 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 20:59:13.859374 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:13.859315 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 20:59:13.915798 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:13.903027 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:59:14.268797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:14.268542 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 20:59:14.358160 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:14.358127 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 20:59:16.143186 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.143147 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz" podUID="1a12c1de-d86f-4748-93e9-b2729d60dbfd" containerName="payload-processing" containerID="cri-o://75e2fc375131a69785f6d8a425b4c3d1a5a9e6247814942f1fb3cc79d1148706" gracePeriod=25 Oct 30 20:59:16.163752 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.163701 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8" podUID="82df227a-4155-4961-be3b-a569fef21692" containerName="payload-pre-processing" containerID="cri-o://ae1b24ef2586df95b6a63cb6f39e0d90edc891adfd6be7d5f006bc3ca8d1b4f9" gracePeriod=25 Oct 30 20:59:16.260018 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.259989 4848 scope.go:122] "RemoveContainer" containerID="58a60bdfad2954ce7a1763b7544f78ffae9cd88796159667a26f7cd0d8d6c3eb" Oct 30 20:59:16.286423 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.286390 4848 generic.go:356] "Generic (PLEG): container finished" podID="1a12c1de-d86f-4748-93e9-b2729d60dbfd" containerID="75e2fc375131a69785f6d8a425b4c3d1a5a9e6247814942f1fb3cc79d1148706" exitCode=0 Oct 30 20:59:16.286548 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.286468 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz" event={"ID":"1a12c1de-d86f-4748-93e9-b2729d60dbfd","Type":"ContainerDied","Data":"75e2fc375131a69785f6d8a425b4c3d1a5a9e6247814942f1fb3cc79d1148706"} Oct 30 20:59:16.289040 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.289013 4848 generic.go:356] "Generic (PLEG): container finished" podID="82df227a-4155-4961-be3b-a569fef21692" containerID="ae1b24ef2586df95b6a63cb6f39e0d90edc891adfd6be7d5f006bc3ca8d1b4f9" exitCode=0 Oct 30 20:59:16.289161 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.289075 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8" event={"ID":"82df227a-4155-4961-be3b-a569fef21692","Type":"ContainerDied","Data":"ae1b24ef2586df95b6a63cb6f39e0d90edc891adfd6be7d5f006bc3ca8d1b4f9"} Oct 30 20:59:16.511824 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.511714 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8" Oct 30 20:59:16.543788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.542120 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82df227a-4155-4961-be3b-a569fef21692-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82df227a-4155-4961-be3b-a569fef21692-plugins-config-volume\") pod \"82df227a-4155-4961-be3b-a569fef21692\" (UID: \"82df227a-4155-4961-be3b-a569fef21692\") " Oct 30 20:59:16.543788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.542225 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82df227a-4155-4961-be3b-a569fef21692-kube-api-access-hv7t8\" (UniqueName: \"kubernetes.io/projected/82df227a-4155-4961-be3b-a569fef21692-kube-api-access-hv7t8\") pod \"82df227a-4155-4961-be3b-a569fef21692\" (UID: \"82df227a-4155-4961-be3b-a569fef21692\") " Oct 30 20:59:16.543788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.543291 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82df227a-4155-4961-be3b-a569fef21692-plugins-config-volume" pod "82df227a-4155-4961-be3b-a569fef21692" (UID: "82df227a-4155-4961-be3b-a569fef21692"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:16.546118 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.546090 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82df227a-4155-4961-be3b-a569fef21692-kube-api-access-hv7t8" pod "82df227a-4155-4961-be3b-a569fef21692" (UID: "82df227a-4155-4961-be3b-a569fef21692"). InnerVolumeSpecName "kube-api-access-hv7t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:16.559459 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.559440 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz" Oct 30 20:59:16.643695 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.643657 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a12c1de-d86f-4748-93e9-b2729d60dbfd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a12c1de-d86f-4748-93e9-b2729d60dbfd-plugins-config-volume\") pod \"1a12c1de-d86f-4748-93e9-b2729d60dbfd\" (UID: \"1a12c1de-d86f-4748-93e9-b2729d60dbfd\") " Oct 30 20:59:16.643882 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.643860 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a12c1de-d86f-4748-93e9-b2729d60dbfd-kube-api-access-9c26w\" (UniqueName: \"kubernetes.io/projected/1a12c1de-d86f-4748-93e9-b2729d60dbfd-kube-api-access-9c26w\") pod \"1a12c1de-d86f-4748-93e9-b2729d60dbfd\" (UID: \"1a12c1de-d86f-4748-93e9-b2729d60dbfd\") " Oct 30 20:59:16.644118 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.644090 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a12c1de-d86f-4748-93e9-b2729d60dbfd-plugins-config-volume" pod "1a12c1de-d86f-4748-93e9-b2729d60dbfd" (UID: "1a12c1de-d86f-4748-93e9-b2729d60dbfd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:16.644555 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.644537 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a12c1de-d86f-4748-93e9-b2729d60dbfd-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:16.644608 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.644562 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82df227a-4155-4961-be3b-a569fef21692-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:16.644608 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.644577 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv7t8\" (UniqueName: \"kubernetes.io/projected/82df227a-4155-4961-be3b-a569fef21692-kube-api-access-hv7t8\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:16.645966 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.645945 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a12c1de-d86f-4748-93e9-b2729d60dbfd-kube-api-access-9c26w" pod "1a12c1de-d86f-4748-93e9-b2729d60dbfd" (UID: "1a12c1de-d86f-4748-93e9-b2729d60dbfd"). InnerVolumeSpecName "kube-api-access-9c26w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:16.750358 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.750325 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c26w\" (UniqueName: \"kubernetes.io/projected/1a12c1de-d86f-4748-93e9-b2729d60dbfd-kube-api-access-9c26w\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:16.807585 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.807505 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m"] Oct 30 20:59:16.808479 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.808462 4848 cpu_manager.go:414] "containerMap: removing container" podUID="82df227a-4155-4961-be3b-a569fef21692" containerName="payload-pre-processing" Oct 30 20:59:16.808479 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.808480 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82df227a-4155-4961-be3b-a569fef21692" containerName="payload-pre-processing" Oct 30 20:59:16.808622 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.808550 4848 cpu_manager.go:414] "containerMap: removing container" podUID="1a12c1de-d86f-4748-93e9-b2729d60dbfd" containerName="payload-processing" Oct 30 20:59:16.808622 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.808557 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a12c1de-d86f-4748-93e9-b2729d60dbfd" containerName="payload-processing" Oct 30 20:59:16.808716 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.808645 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="82df227a-4155-4961-be3b-a569fef21692" containerName="payload-pre-processing" Oct 30 20:59:16.808716 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.808663 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a12c1de-d86f-4748-93e9-b2729d60dbfd" containerName="payload-processing" Oct 30 20:59:16.812482 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.812462 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:16.824354 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.820950 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b80638-openshift-default-dockercfg-wlxc4\"" Oct 30 20:59:16.828401 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.827367 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m"] Oct 30 20:59:16.859717 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.856901 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcf5t\" (UniqueName: \"kubernetes.io/projected/e9b4f116-4af8-4321-887c-1df07ed0cd5d-kube-api-access-vcf5t\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:16.859717 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.856962 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-podinfo\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:16.859717 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.857036 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-workload-socket\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:16.859717 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.857105 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-data\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:16.859717 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.857134 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-token\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:16.859717 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.857194 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-envoy\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:16.859717 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.857272 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-credential-socket\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:16.859717 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.857309 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istiod-ca-cert\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:16.859717 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.857380 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-workload-certs\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:16.984813 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.979465 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-credential-socket\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:16.986157 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.985785 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-credential-socket\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:16.987160 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.987098 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istiod-ca-cert\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:16.987356 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.987337 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istiod-ca-cert\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:16.988196 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.988131 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-workload-certs\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:17.002450 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.987757 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-workload-certs\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:17.002450 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.988654 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcf5t\" (UniqueName: \"kubernetes.io/projected/e9b4f116-4af8-4321-887c-1df07ed0cd5d-kube-api-access-vcf5t\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:17.002450 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.988702 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-podinfo\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:17.002450 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.988736 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-workload-socket\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:17.002450 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.988837 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-data\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:17.002450 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.988864 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-token\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:17.002450 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:16.988931 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-envoy\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:17.004306 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.003522 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-workload-socket\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:17.004862 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.004816 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-data\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:17.101853 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.101044 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h" podUID="1447449b-6f3f-4cde-9a36-0f56ce7a8b16" containerName="payload-processing" containerID="cri-o://96a726b3b21406d49105f334a20dd32f89fad19a85b11b87a9de333abe75cd5c" gracePeriod=25 Oct 30 20:59:17.108844 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.103788 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-podinfo\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:17.122100 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.122045 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b" podUID="6b3101d0-f69c-47ef-b438-6fc1966e913b" containerName="payload-pre-processing" containerID="cri-o://9ca4d934c73bfc206688163a956f6a6f4cc535dd5781df899bb168cb33372915" gracePeriod=25 Oct 30 20:59:17.128057 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.125721 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcf5t\" (UniqueName: \"kubernetes.io/projected/e9b4f116-4af8-4321-887c-1df07ed0cd5d-kube-api-access-vcf5t\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:17.132993 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.132912 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-token\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:17.145897 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.145862 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-envoy\") pod \"e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:17.178759 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.157631 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:17.325797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.324917 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz" event={"ID":"1a12c1de-d86f-4748-93e9-b2729d60dbfd","Type":"ContainerDied","Data":"bfb375bef79042bda32b93b6649c2c5803fb07866ce770a4817cc13c09a64cd4"} Oct 30 20:59:17.325797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.325003 4848 scope.go:122] "RemoveContainer" containerID="75e2fc375131a69785f6d8a425b4c3d1a5a9e6247814942f1fb3cc79d1148706" Oct 30 20:59:17.325797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.325181 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz" Oct 30 20:59:17.329814 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.329622 4848 generic.go:356] "Generic (PLEG): container finished" podID="6b3101d0-f69c-47ef-b438-6fc1966e913b" containerID="9ca4d934c73bfc206688163a956f6a6f4cc535dd5781df899bb168cb33372915" exitCode=0 Oct 30 20:59:17.329814 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.329711 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b" event={"ID":"6b3101d0-f69c-47ef-b438-6fc1966e913b","Type":"ContainerDied","Data":"9ca4d934c73bfc206688163a956f6a6f4cc535dd5781df899bb168cb33372915"} Oct 30 20:59:17.333119 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.332856 4848 generic.go:356] "Generic (PLEG): container finished" podID="1447449b-6f3f-4cde-9a36-0f56ce7a8b16" containerID="96a726b3b21406d49105f334a20dd32f89fad19a85b11b87a9de333abe75cd5c" exitCode=0 Oct 30 20:59:17.333119 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.333036 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h" event={"ID":"1447449b-6f3f-4cde-9a36-0f56ce7a8b16","Type":"ContainerDied","Data":"96a726b3b21406d49105f334a20dd32f89fad19a85b11b87a9de333abe75cd5c"} Oct 30 20:59:17.337043 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.335728 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8" event={"ID":"82df227a-4155-4961-be3b-a569fef21692","Type":"ContainerDied","Data":"191376f85c81fbfc4681fb94846be143b0a45d5330de5d3c501820ec050eda3f"} Oct 30 20:59:17.337043 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.335883 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8" Oct 30 20:59:17.363733 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.363618 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8"] Oct 30 20:59:17.366068 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.365923 4848 scope.go:122] "RemoveContainer" containerID="ae1b24ef2586df95b6a63cb6f39e0d90edc891adfd6be7d5f006bc3ca8d1b4f9" Oct 30 20:59:17.369888 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.369815 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-2b7ca7-6569654dff-8gvz8"] Oct 30 20:59:17.384685 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.384646 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz"] Oct 30 20:59:17.391007 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.390979 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-2b7ca7-84d5cc7bc-hqnwz"] Oct 30 20:59:17.499839 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.498252 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m"] Oct 30 20:59:17.519734 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:59:17.519625 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b4f116_4af8_4321_887c_1df07ed0cd5d.slice/crio-64e017c899de594f9630513f5a7cba7c37e09c4a76343108cdb8a7093480da7e WatchSource:0}: Error finding container 64e017c899de594f9630513f5a7cba7c37e09c4a76343108cdb8a7093480da7e: Status 404 returned error can't find the container with id 64e017c899de594f9630513f5a7cba7c37e09c4a76343108cdb8a7093480da7e Oct 30 20:59:17.621021 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.616425 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng"] Oct 30 20:59:17.629909 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.629870 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:17.630042 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.629961 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:17.630042 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.630032 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:17.645975 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.645894 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h" Oct 30 20:59:17.667783 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.667072 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b" Oct 30 20:59:17.719421 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.717281 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3101d0-f69c-47ef-b438-6fc1966e913b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3101d0-f69c-47ef-b438-6fc1966e913b-plugins-config-volume\") pod \"6b3101d0-f69c-47ef-b438-6fc1966e913b\" (UID: \"6b3101d0-f69c-47ef-b438-6fc1966e913b\") " Oct 30 20:59:17.719421 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.717326 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3101d0-f69c-47ef-b438-6fc1966e913b-kube-api-access-96jqg\" (UniqueName: \"kubernetes.io/projected/6b3101d0-f69c-47ef-b438-6fc1966e913b-kube-api-access-96jqg\") pod \"6b3101d0-f69c-47ef-b438-6fc1966e913b\" (UID: \"6b3101d0-f69c-47ef-b438-6fc1966e913b\") " Oct 30 20:59:17.719421 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.717427 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1447449b-6f3f-4cde-9a36-0f56ce7a8b16-kube-api-access-582bl\" (UniqueName: \"kubernetes.io/projected/1447449b-6f3f-4cde-9a36-0f56ce7a8b16-kube-api-access-582bl\") pod \"1447449b-6f3f-4cde-9a36-0f56ce7a8b16\" (UID: \"1447449b-6f3f-4cde-9a36-0f56ce7a8b16\") " Oct 30 20:59:17.719421 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.717481 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1447449b-6f3f-4cde-9a36-0f56ce7a8b16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1447449b-6f3f-4cde-9a36-0f56ce7a8b16-plugins-config-volume\") pod \"1447449b-6f3f-4cde-9a36-0f56ce7a8b16\" (UID: \"1447449b-6f3f-4cde-9a36-0f56ce7a8b16\") " Oct 30 20:59:17.719421 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.717704 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3101d0-f69c-47ef-b438-6fc1966e913b-plugins-config-volume" pod "6b3101d0-f69c-47ef-b438-6fc1966e913b" (UID: "6b3101d0-f69c-47ef-b438-6fc1966e913b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:17.719421 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.718287 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b3101d0-f69c-47ef-b438-6fc1966e913b-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:17.722385 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.722328 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1447449b-6f3f-4cde-9a36-0f56ce7a8b16-kube-api-access-582bl" pod "1447449b-6f3f-4cde-9a36-0f56ce7a8b16" (UID: "1447449b-6f3f-4cde-9a36-0f56ce7a8b16"). InnerVolumeSpecName "kube-api-access-582bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:17.722532 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.722493 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1447449b-6f3f-4cde-9a36-0f56ce7a8b16-plugins-config-volume" pod "1447449b-6f3f-4cde-9a36-0f56ce7a8b16" (UID: "1447449b-6f3f-4cde-9a36-0f56ce7a8b16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:17.729586 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.729487 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3101d0-f69c-47ef-b438-6fc1966e913b-kube-api-access-96jqg" pod "6b3101d0-f69c-47ef-b438-6fc1966e913b" (UID: "6b3101d0-f69c-47ef-b438-6fc1966e913b"). InnerVolumeSpecName "kube-api-access-96jqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:17.820491 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.820411 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-582bl\" (UniqueName: \"kubernetes.io/projected/1447449b-6f3f-4cde-9a36-0f56ce7a8b16-kube-api-access-582bl\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:17.820491 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.820448 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1447449b-6f3f-4cde-9a36-0f56ce7a8b16-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:17.820491 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:17.820464 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96jqg\" (UniqueName: \"kubernetes.io/projected/6b3101d0-f69c-47ef-b438-6fc1966e913b-kube-api-access-96jqg\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:18.351760 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:18.351726 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b" event={"ID":"6b3101d0-f69c-47ef-b438-6fc1966e913b","Type":"ContainerDied","Data":"c451e946eba18a3b90b6f1975277a9b189e9cab0d110a2ea29bfbc71eba37d25"} Oct 30 20:59:18.352236 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:18.352218 4848 scope.go:122] "RemoveContainer" containerID="9ca4d934c73bfc206688163a956f6a6f4cc535dd5781df899bb168cb33372915" Oct 30 20:59:18.352435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:18.352422 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b" Oct 30 20:59:18.361348 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:18.361322 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" event={"ID":"e9b4f116-4af8-4321-887c-1df07ed0cd5d","Type":"ContainerStarted","Data":"cfa3bcfdab31604d7d29849a2da76015971a05de5efb266754a476a3b0e5fab7"} Oct 30 20:59:18.363153 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:18.363134 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" event={"ID":"e9b4f116-4af8-4321-887c-1df07ed0cd5d","Type":"ContainerStarted","Data":"64e017c899de594f9630513f5a7cba7c37e09c4a76343108cdb8a7093480da7e"} Oct 30 20:59:18.364438 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:18.364408 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h" event={"ID":"1447449b-6f3f-4cde-9a36-0f56ce7a8b16","Type":"ContainerDied","Data":"6073d3e727c27f29a14d9d1c27ee437160b14d3e0eb3d56e8bfa8e06ba6ef9f3"} Oct 30 20:59:18.364543 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:18.364454 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h" Oct 30 20:59:18.371915 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:18.371850 4848 scope.go:122] "RemoveContainer" containerID="96a726b3b21406d49105f334a20dd32f89fad19a85b11b87a9de333abe75cd5c" Oct 30 20:59:18.402630 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:18.402543 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" podStartSLOduration=2.402524237 podStartE2EDuration="2.402524237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:18.385195168 +0000 UTC m=+1263.565091707" watchObservedRunningTime="2026-10-30 20:59:18.402524237 +0000 UTC m=+1263.582420746" Oct 30 20:59:18.402884 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:18.402792 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b"] Oct 30 20:59:18.407366 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:18.407333 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-38c5ca-78b9ffd675-lpk4b"] Oct 30 20:59:18.432086 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:18.432045 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h"] Oct 30 20:59:18.438245 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:18.438216 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-38c5ca-659c76c7df-bp65h"] Oct 30 20:59:18.996511 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:18.996461 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1447449b-6f3f-4cde-9a36-0f56ce7a8b16" path="/var/lib/kubelet/pods/1447449b-6f3f-4cde-9a36-0f56ce7a8b16/volumes" Oct 30 20:59:18.997111 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:18.996944 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a12c1de-d86f-4748-93e9-b2729d60dbfd" path="/var/lib/kubelet/pods/1a12c1de-d86f-4748-93e9-b2729d60dbfd/volumes" Oct 30 20:59:19.000168 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:18.997574 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b3101d0-f69c-47ef-b438-6fc1966e913b" path="/var/lib/kubelet/pods/6b3101d0-f69c-47ef-b438-6fc1966e913b/volumes" Oct 30 20:59:19.000168 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:18.998167 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82df227a-4155-4961-be3b-a569fef21692" path="/var/lib/kubelet/pods/82df227a-4155-4961-be3b-a569fef21692/volumes" Oct 30 20:59:19.159083 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:19.159048 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:19.165851 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:19.165818 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:19.376880 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:19.376804 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:19.377892 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:19.377873 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 20:59:21.888804 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:21.888748 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw"] Oct 30 20:59:21.889456 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:21.889439 4848 cpu_manager.go:414] "containerMap: removing container" podUID="6b3101d0-f69c-47ef-b438-6fc1966e913b" containerName="payload-pre-processing" Oct 30 20:59:21.889512 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:21.889457 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3101d0-f69c-47ef-b438-6fc1966e913b" containerName="payload-pre-processing" Oct 30 20:59:21.889512 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:21.889469 4848 cpu_manager.go:414] "containerMap: removing container" podUID="1447449b-6f3f-4cde-9a36-0f56ce7a8b16" containerName="payload-processing" Oct 30 20:59:21.889512 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:21.889474 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1447449b-6f3f-4cde-9a36-0f56ce7a8b16" containerName="payload-processing" Oct 30 20:59:21.889607 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:21.889583 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3101d0-f69c-47ef-b438-6fc1966e913b" containerName="payload-pre-processing" Oct 30 20:59:21.889607 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:21.889591 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="1447449b-6f3f-4cde-9a36-0f56ce7a8b16" containerName="payload-processing" Oct 30 20:59:21.927416 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:21.927386 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw"] Oct 30 20:59:21.927580 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:21.927523 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw" Oct 30 20:59:21.936632 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:21.936603 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b80638\"" Oct 30 20:59:22.036463 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:22.036423 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8049a5ba-174e-4576-9adb-9c4eac65f0b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw\" (UID: \"8049a5ba-174e-4576-9adb-9c4eac65f0b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw" Oct 30 20:59:22.049907 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:22.049867 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-644c5\" (UniqueName: \"kubernetes.io/projected/8049a5ba-174e-4576-9adb-9c4eac65f0b5-kube-api-access-644c5\") pod \"payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw\" (UID: \"8049a5ba-174e-4576-9adb-9c4eac65f0b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw" Oct 30 20:59:22.160900 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:22.155062 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8049a5ba-174e-4576-9adb-9c4eac65f0b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw\" (UID: \"8049a5ba-174e-4576-9adb-9c4eac65f0b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw" Oct 30 20:59:22.160900 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:22.155120 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-644c5\" (UniqueName: \"kubernetes.io/projected/8049a5ba-174e-4576-9adb-9c4eac65f0b5-kube-api-access-644c5\") pod \"payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw\" (UID: \"8049a5ba-174e-4576-9adb-9c4eac65f0b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw" Oct 30 20:59:22.160900 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:22.156068 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8049a5ba-174e-4576-9adb-9c4eac65f0b5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw\" (UID: \"8049a5ba-174e-4576-9adb-9c4eac65f0b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw" Oct 30 20:59:22.182786 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:22.182228 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-644c5\" (UniqueName: \"kubernetes.io/projected/8049a5ba-174e-4576-9adb-9c4eac65f0b5-kube-api-access-644c5\") pod \"payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw\" (UID: \"8049a5ba-174e-4576-9adb-9c4eac65f0b5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw" Oct 30 20:59:22.279788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:22.278834 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw" Oct 30 20:59:22.511197 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:59:22.511155 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:59:22.589181 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:22.589142 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw"] Oct 30 20:59:22.590026 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:59:22.589994 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8049a5ba_174e_4576_9adb_9c4eac65f0b5.slice/crio-fef0c7150441bbb5944a31b28635a1d78402a48a21c33491ed9c63283d336af8 WatchSource:0}: Error finding container fef0c7150441bbb5944a31b28635a1d78402a48a21c33491ed9c63283d336af8: Status 404 returned error can't find the container with id fef0c7150441bbb5944a31b28635a1d78402a48a21c33491ed9c63283d336af8 Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:22.617266 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng" podUID="c7f6682a-f8be-4abf-99e7-0977b3e61600" containerName="payload-processing" containerID="cri-o://025b78c62bbc38e363008902b99ce8530ddb409ad8d5dbcd10c1422ad4cba4e3" gracePeriod=25 Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:22.932323 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng" Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:22.966363 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f6682a-f8be-4abf-99e7-0977b3e61600-kube-api-access-vkxz9\" (UniqueName: \"kubernetes.io/projected/c7f6682a-f8be-4abf-99e7-0977b3e61600-kube-api-access-vkxz9\") pod \"c7f6682a-f8be-4abf-99e7-0977b3e61600\" (UID: \"c7f6682a-f8be-4abf-99e7-0977b3e61600\") " Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:22.966637 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f6682a-f8be-4abf-99e7-0977b3e61600-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f6682a-f8be-4abf-99e7-0977b3e61600-plugins-config-volume\") pod \"c7f6682a-f8be-4abf-99e7-0977b3e61600\" (UID: \"c7f6682a-f8be-4abf-99e7-0977b3e61600\") " Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:22.967065 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f6682a-f8be-4abf-99e7-0977b3e61600-plugins-config-volume" pod "c7f6682a-f8be-4abf-99e7-0977b3e61600" (UID: "c7f6682a-f8be-4abf-99e7-0977b3e61600"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:22.967344 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f6682a-f8be-4abf-99e7-0977b3e61600-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:22.968605 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f6682a-f8be-4abf-99e7-0977b3e61600-kube-api-access-vkxz9" pod "c7f6682a-f8be-4abf-99e7-0977b3e61600" (UID: "c7f6682a-f8be-4abf-99e7-0977b3e61600"). InnerVolumeSpecName "kube-api-access-vkxz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:23.068879 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkxz9\" (UniqueName: \"kubernetes.io/projected/c7f6682a-f8be-4abf-99e7-0977b3e61600-kube-api-access-vkxz9\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:23.411153 4848 generic.go:356] "Generic (PLEG): container finished" podID="c7f6682a-f8be-4abf-99e7-0977b3e61600" containerID="025b78c62bbc38e363008902b99ce8530ddb409ad8d5dbcd10c1422ad4cba4e3" exitCode=0 Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:23.411221 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng" Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:23.411244 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng" event={"ID":"c7f6682a-f8be-4abf-99e7-0977b3e61600","Type":"ContainerDied","Data":"025b78c62bbc38e363008902b99ce8530ddb409ad8d5dbcd10c1422ad4cba4e3"} Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:23.411297 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng" event={"ID":"c7f6682a-f8be-4abf-99e7-0977b3e61600","Type":"ContainerDied","Data":"2b18076e8628f0aeb7ab1b9f60bcd3603dff3dfea4a3ba4060ce8c4abc35ec2e"} Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:23.411326 4848 scope.go:122] "RemoveContainer" containerID="025b78c62bbc38e363008902b99ce8530ddb409ad8d5dbcd10c1422ad4cba4e3" Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:23.413362 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw" event={"ID":"8049a5ba-174e-4576-9adb-9c4eac65f0b5","Type":"ContainerStarted","Data":"02c402196b36d3e2028997424efa29ac6206f812dd784d233156fb07f3717d0f"} Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:23.413407 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw" event={"ID":"8049a5ba-174e-4576-9adb-9c4eac65f0b5","Type":"ContainerStarted","Data":"fef0c7150441bbb5944a31b28635a1d78402a48a21c33491ed9c63283d336af8"} Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:23.413452 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw" Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:23.422028 4848 scope.go:122] "RemoveContainer" containerID="025b78c62bbc38e363008902b99ce8530ddb409ad8d5dbcd10c1422ad4cba4e3" Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:59:23.422276 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"025b78c62bbc38e363008902b99ce8530ddb409ad8d5dbcd10c1422ad4cba4e3\": container with ID starting with 025b78c62bbc38e363008902b99ce8530ddb409ad8d5dbcd10c1422ad4cba4e3 not found: ID does not exist" containerID="025b78c62bbc38e363008902b99ce8530ddb409ad8d5dbcd10c1422ad4cba4e3" Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:23.422298 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"025b78c62bbc38e363008902b99ce8530ddb409ad8d5dbcd10c1422ad4cba4e3"} err="failed to get container status \"025b78c62bbc38e363008902b99ce8530ddb409ad8d5dbcd10c1422ad4cba4e3\": rpc error: code = NotFound desc = could not find container \"025b78c62bbc38e363008902b99ce8530ddb409ad8d5dbcd10c1422ad4cba4e3\": container with ID starting with 025b78c62bbc38e363008902b99ce8530ddb409ad8d5dbcd10c1422ad4cba4e3 not found: ID does not exist" Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:23.454686 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw" podStartSLOduration=2.454670899 podStartE2EDuration="2.454670899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:23.453467989 +0000 UTC m=+1268.633364496" watchObservedRunningTime="2026-10-30 20:59:23.454670899 +0000 UTC m=+1268.634567406" Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:23.510699 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng"] Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:23.516129 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-7bbb764d6c-pzmng"] Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:23.748953 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d"] Oct 30 20:59:23.832597 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:23.749232 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" podUID="2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2" containerName="maas-api" containerID="cri-o://718b4189c1379f6f2040ef44aa78ba99b7729dbaed5eb9e436e5f9a352974246" gracePeriod=30 Oct 30 20:59:23.936219 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:23.936194 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh"] Oct 30 20:59:24.189238 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.184813 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz"] Oct 30 20:59:24.315376 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.315351 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" Oct 30 20:59:24.421508 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.421419 4848 generic.go:356] "Generic (PLEG): container finished" podID="2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2" containerID="718b4189c1379f6f2040ef44aa78ba99b7729dbaed5eb9e436e5f9a352974246" exitCode=0 Oct 30 20:59:24.421508 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.421493 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" Oct 30 20:59:24.421787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.421503 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" event={"ID":"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2","Type":"ContainerDied","Data":"718b4189c1379f6f2040ef44aa78ba99b7729dbaed5eb9e436e5f9a352974246"} Oct 30 20:59:24.421787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.421601 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d" event={"ID":"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2","Type":"ContainerDied","Data":"329d351e9142d7a7689ebfcc65a9fe9ff6de32be626471612cde2914c01fba07"} Oct 30 20:59:24.421787 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.421633 4848 scope.go:122] "RemoveContainer" containerID="718b4189c1379f6f2040ef44aa78ba99b7729dbaed5eb9e436e5f9a352974246" Oct 30 20:59:24.423732 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.423705 4848 generic.go:356] "Generic (PLEG): container finished" podID="1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" containerID="2d358be9ef0986f7d0e7776ea57191f5e2a4aa52162894a3138941fb7128cceb" exitCode=0 Oct 30 20:59:24.423855 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.423796 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" event={"ID":"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e","Type":"ContainerDied","Data":"2d358be9ef0986f7d0e7776ea57191f5e2a4aa52162894a3138941fb7128cceb"} Oct 30 20:59:24.424242 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.424226 4848 scope.go:122] "RemoveContainer" containerID="2d358be9ef0986f7d0e7776ea57191f5e2a4aa52162894a3138941fb7128cceb" Oct 30 20:59:24.425613 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.425588 4848 generic.go:356] "Generic (PLEG): container finished" podID="b0998438-c7f6-4cb0-ba25-b406d43b8959" containerID="65b26dc1623be9f3195ac4dbca57546a9696e367a27f16f7eb8099116c3df05d" exitCode=0 Oct 30 20:59:24.425828 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.425806 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" event={"ID":"b0998438-c7f6-4cb0-ba25-b406d43b8959","Type":"ContainerDied","Data":"65b26dc1623be9f3195ac4dbca57546a9696e367a27f16f7eb8099116c3df05d"} Oct 30 20:59:24.426143 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.426086 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:24.426225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.426203 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:24.426289 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.426236 4848 scope.go:122] "RemoveContainer" containerID="65b26dc1623be9f3195ac4dbca57546a9696e367a27f16f7eb8099116c3df05d" Oct 30 20:59:24.426289 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.426246 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:24.427733 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.427706 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:24.427840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.427782 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:24.427840 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.427830 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:24.428482 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.428464 4848 generic.go:356] "Generic (PLEG): container finished" podID="5c00d1f3-b911-40fd-8772-3ed4bba10d75" containerID="26f56a772ac1cb31314790b81e470d8800e6e09c9790f2c14135946a6b168088" exitCode=0 Oct 30 20:59:24.428544 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.428526 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" event={"ID":"5c00d1f3-b911-40fd-8772-3ed4bba10d75","Type":"ContainerDied","Data":"26f56a772ac1cb31314790b81e470d8800e6e09c9790f2c14135946a6b168088"} Oct 30 20:59:24.428924 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.428902 4848 scope.go:122] "RemoveContainer" containerID="26f56a772ac1cb31314790b81e470d8800e6e09c9790f2c14135946a6b168088" Oct 30 20:59:24.434060 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.434037 4848 scope.go:122] "RemoveContainer" containerID="718b4189c1379f6f2040ef44aa78ba99b7729dbaed5eb9e436e5f9a352974246" Oct 30 20:59:24.434245 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.434219 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:24.434304 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.434291 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:24.434360 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.434337 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:24.434422 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:59:24.434351 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"718b4189c1379f6f2040ef44aa78ba99b7729dbaed5eb9e436e5f9a352974246\": container with ID starting with 718b4189c1379f6f2040ef44aa78ba99b7729dbaed5eb9e436e5f9a352974246 not found: ID does not exist" containerID="718b4189c1379f6f2040ef44aa78ba99b7729dbaed5eb9e436e5f9a352974246" Oct 30 20:59:24.434422 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.434384 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"718b4189c1379f6f2040ef44aa78ba99b7729dbaed5eb9e436e5f9a352974246"} err="failed to get container status \"718b4189c1379f6f2040ef44aa78ba99b7729dbaed5eb9e436e5f9a352974246\": rpc error: code = NotFound desc = could not find container \"718b4189c1379f6f2040ef44aa78ba99b7729dbaed5eb9e436e5f9a352974246\": container with ID starting with 718b4189c1379f6f2040ef44aa78ba99b7729dbaed5eb9e436e5f9a352974246 not found: ID does not exist" Oct 30 20:59:24.505902 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.498149 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-openshift-service-ca\") pod \"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2\" (UID: \"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2\") " Oct 30 20:59:24.505902 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.498309 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-kube-api-access-28m2w\" (UniqueName: \"kubernetes.io/projected/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-kube-api-access-28m2w\") pod \"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2\" (UID: \"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2\") " Oct 30 20:59:24.505902 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.498397 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-maas-api-tls\") pod \"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2\" (UID: \"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2\") " Oct 30 20:59:24.505902 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.498539 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-metrics-certs\") pod \"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2\" (UID: \"2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2\") " Oct 30 20:59:24.576314 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.576271 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-openshift-service-ca" pod "2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2" (UID: "2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:24.626683 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.618259 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:24.648170 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.647833 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-kube-api-access-28m2w" pod "2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2" (UID: "2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2"). InnerVolumeSpecName "kube-api-access-28m2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:24.648170 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.647902 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-metrics-certs" pod "2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2" (UID: "2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:24.657411 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.656825 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-maas-api-tls" pod "2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2" (UID: "2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:24.727805 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.726954 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:24.728123 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.726982 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28m2w\" (UniqueName: \"kubernetes.io/projected/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-kube-api-access-28m2w\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:24.728123 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.727990 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:24.776241 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.776206 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d"] Oct 30 20:59:24.786797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.783541 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2b7ca7-bfddfd446-wnc7d"] Oct 30 20:59:24.944451 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.944371 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2" path="/var/lib/kubelet/pods/2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2/volumes" Oct 30 20:59:24.945551 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:24.945533 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f6682a-f8be-4abf-99e7-0977b3e61600" path="/var/lib/kubelet/pods/c7f6682a-f8be-4abf-99e7-0977b3e61600/volumes" Oct 30 20:59:25.057881 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:25.057715 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:59:25.467824 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:25.466133 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" event={"ID":"5c00d1f3-b911-40fd-8772-3ed4bba10d75","Type":"ContainerStarted","Data":"e9d93d1afae7368d5a19cd9de7b9d67739392239a1ec0ab41647d36196a89602"} Oct 30 20:59:25.469560 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:25.469531 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" event={"ID":"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e","Type":"ContainerStarted","Data":"fca5ab014eb2c4e1fa7839c033686750beba009607080d89085b579f8089d104"} Oct 30 20:59:25.481435 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:25.478885 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" event={"ID":"b0998438-c7f6-4cb0-ba25-b406d43b8959","Type":"ContainerStarted","Data":"6cf00af8879dcc3d0bbd9a3d31607986495738ceaf18332628ccac0cf8af7f26"} Oct 30 20:59:25.639007 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:25.638975 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:59:25.642590 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:25.642559 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:59:25.642879 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:25.642845 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" podUID="1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 20:59:25.657905 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:25.657873 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:59:25.658329 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:25.658272 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:59:25.708949 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:25.708920 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:59:25.719985 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:25.719891 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 20:59:25.720246 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:25.720206 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" podUID="b0998438-c7f6-4cb0-ba25-b406d43b8959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 20:59:26.137224 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:26.137132 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:59:26.138234 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:26.138207 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 20:59:26.138421 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:26.138398 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" podUID="5c00d1f3-b911-40fd-8772-3ed4bba10d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 20:59:26.660120 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:26.660059 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:59:26.660314 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:26.660174 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" podUID="1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 20:59:26.801948 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:26.801905 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 20:59:26.802158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:26.801985 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" podUID="b0998438-c7f6-4cb0-ba25-b406d43b8959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 20:59:27.133065 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:27.133026 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 20:59:27.172223 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:27.172177 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" podUID="5c00d1f3-b911-40fd-8772-3ed4bba10d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 20:59:27.642611 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:27.642143 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:59:27.642611 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:27.642208 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" podUID="1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 20:59:27.726797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:27.726307 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 20:59:27.726797 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:27.726391 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" podUID="b0998438-c7f6-4cb0-ba25-b406d43b8959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 20:59:28.137790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:28.137308 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 20:59:28.137790 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:28.137368 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" podUID="5c00d1f3-b911-40fd-8772-3ed4bba10d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 20:59:28.660791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:28.642148 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:59:28.660791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:28.642231 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" podUID="1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 20:59:28.710434 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:28.707166 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 20:59:28.710434 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:28.707245 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" podUID="b0998438-c7f6-4cb0-ba25-b406d43b8959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 20:59:28.945893 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:28.945841 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh" podUID="9ac51789-e545-4331-ae29-f99e8150fc15" containerName="payload-processing" containerID="cri-o://1b5b30e87296d1d849d232dd0af5765e48ddedd6bb10cbb2a12f30862aaff0c8" gracePeriod=25 Oct 30 20:59:29.156676 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:29.156621 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 20:59:29.161742 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:29.161704 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" podUID="5c00d1f3-b911-40fd-8772-3ed4bba10d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 20:59:29.188087 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:29.187869 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz" podUID="9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d" containerName="payload-pre-processing" containerID="cri-o://c1cd4f243d2e2df8807d47c93c4b55f3c123bb7fe620cb135eee197f3f466745" gracePeriod=25 Oct 30 20:59:29.607871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:29.603079 4848 generic.go:356] "Generic (PLEG): container finished" podID="9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d" containerID="c1cd4f243d2e2df8807d47c93c4b55f3c123bb7fe620cb135eee197f3f466745" exitCode=0 Oct 30 20:59:29.607871 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:29.603148 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz" event={"ID":"9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d","Type":"ContainerDied","Data":"c1cd4f243d2e2df8807d47c93c4b55f3c123bb7fe620cb135eee197f3f466745"} Oct 30 20:59:29.663869 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:29.662846 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:59:29.663869 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:29.662916 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" podUID="1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 20:59:29.696784 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:29.695965 4848 generic.go:356] "Generic (PLEG): container finished" podID="9ac51789-e545-4331-ae29-f99e8150fc15" containerID="1b5b30e87296d1d849d232dd0af5765e48ddedd6bb10cbb2a12f30862aaff0c8" exitCode=0 Oct 30 20:59:29.696784 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:29.696459 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh" event={"ID":"9ac51789-e545-4331-ae29-f99e8150fc15","Type":"ContainerDied","Data":"1b5b30e87296d1d849d232dd0af5765e48ddedd6bb10cbb2a12f30862aaff0c8"} Oct 30 20:59:29.803192 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:29.801573 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 20:59:29.803192 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:29.801631 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" podUID="b0998438-c7f6-4cb0-ba25-b406d43b8959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 20:59:29.981464 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:29.925109 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh" Oct 30 20:59:30.136806 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.133437 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ac51789-e545-4331-ae29-f99e8150fc15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac51789-e545-4331-ae29-f99e8150fc15-plugins-config-volume\") pod \"9ac51789-e545-4331-ae29-f99e8150fc15\" (UID: \"9ac51789-e545-4331-ae29-f99e8150fc15\") " Oct 30 20:59:30.136806 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.133503 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac51789-e545-4331-ae29-f99e8150fc15-kube-api-access-v5pvx\" (UniqueName: \"kubernetes.io/projected/9ac51789-e545-4331-ae29-f99e8150fc15-kube-api-access-v5pvx\") pod \"9ac51789-e545-4331-ae29-f99e8150fc15\" (UID: \"9ac51789-e545-4331-ae29-f99e8150fc15\") " Oct 30 20:59:30.167811 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.144043 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 20:59:30.167811 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.144104 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" podUID="5c00d1f3-b911-40fd-8772-3ed4bba10d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 20:59:30.171060 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.171012 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac51789-e545-4331-ae29-f99e8150fc15-plugins-config-volume" pod "9ac51789-e545-4331-ae29-f99e8150fc15" (UID: "9ac51789-e545-4331-ae29-f99e8150fc15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:30.211792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.194835 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn"] Oct 30 20:59:30.215789 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.212207 4848 cpu_manager.go:414] "containerMap: removing container" podUID="c7f6682a-f8be-4abf-99e7-0977b3e61600" containerName="payload-processing" Oct 30 20:59:30.238776 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.238734 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f6682a-f8be-4abf-99e7-0977b3e61600" containerName="payload-processing" Oct 30 20:59:30.238963 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.238947 4848 cpu_manager.go:414] "containerMap: removing container" podUID="2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2" containerName="maas-api" Oct 30 20:59:30.239062 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.239051 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2" containerName="maas-api" Oct 30 20:59:30.244919 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.244894 4848 cpu_manager.go:414] "containerMap: removing container" podUID="9ac51789-e545-4331-ae29-f99e8150fc15" containerName="payload-processing" Oct 30 20:59:30.327994 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.252801 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac51789-e545-4331-ae29-f99e8150fc15" containerName="payload-processing" Oct 30 20:59:30.327994 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.253055 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f6682a-f8be-4abf-99e7-0977b3e61600" containerName="payload-processing" Oct 30 20:59:30.327994 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.253078 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab4ad8f-4f9f-4b3c-a7d8-f8b0af3b5eb2" containerName="maas-api" Oct 30 20:59:30.327994 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.253101 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac51789-e545-4331-ae29-f99e8150fc15" containerName="payload-processing" Oct 30 20:59:30.327994 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.262062 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ac51789-e545-4331-ae29-f99e8150fc15-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:30.389557 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.389520 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn"] Oct 30 20:59:30.389830 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.389811 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48"] Oct 30 20:59:30.408759 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.408716 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac51789-e545-4331-ae29-f99e8150fc15-kube-api-access-v5pvx" pod "9ac51789-e545-4331-ae29-f99e8150fc15" (UID: "9ac51789-e545-4331-ae29-f99e8150fc15"). InnerVolumeSpecName "kube-api-access-v5pvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:30.475233 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.475192 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn" Oct 30 20:59:30.487226 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.487186 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e19f2d2-409b-457e-8f87-d4308b2c53ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn\" (UID: \"4e19f2d2-409b-457e-8f87-d4308b2c53ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn" Oct 30 20:59:30.487483 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.487333 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9rhc\" (UniqueName: \"kubernetes.io/projected/4e19f2d2-409b-457e-8f87-d4308b2c53ff-kube-api-access-h9rhc\") pod \"payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn\" (UID: \"4e19f2d2-409b-457e-8f87-d4308b2c53ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn" Oct 30 20:59:30.516854 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.488323 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5pvx\" (UniqueName: \"kubernetes.io/projected/9ac51789-e545-4331-ae29-f99e8150fc15-kube-api-access-v5pvx\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:30.517057 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.493849 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b46fb7-dockercfg-jq5sw\"" Oct 30 20:59:30.526091 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.526060 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b46fb7\"" Oct 30 20:59:30.823642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.631609 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e19f2d2-409b-457e-8f87-d4308b2c53ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn\" (UID: \"4e19f2d2-409b-457e-8f87-d4308b2c53ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn" Oct 30 20:59:30.823642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.631742 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9rhc\" (UniqueName: \"kubernetes.io/projected/4e19f2d2-409b-457e-8f87-d4308b2c53ff-kube-api-access-h9rhc\") pod \"payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn\" (UID: \"4e19f2d2-409b-457e-8f87-d4308b2c53ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn" Oct 30 20:59:30.823642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.634560 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e19f2d2-409b-457e-8f87-d4308b2c53ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn\" (UID: \"4e19f2d2-409b-457e-8f87-d4308b2c53ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn" Oct 30 20:59:30.823642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.646296 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9rhc\" (UniqueName: \"kubernetes.io/projected/4e19f2d2-409b-457e-8f87-d4308b2c53ff-kube-api-access-h9rhc\") pod \"payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn\" (UID: \"4e19f2d2-409b-457e-8f87-d4308b2c53ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn" Oct 30 20:59:30.823642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.669659 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:59:30.823642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.669879 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" podUID="1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 20:59:30.823642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.709785 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 20:59:30.823642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.709889 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" podUID="b0998438-c7f6-4cb0-ba25-b406d43b8959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 20:59:30.823642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.821233 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48" Oct 30 20:59:30.951029 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.870699 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz" Oct 30 20:59:30.951029 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:30.889091 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48"] Oct 30 20:59:31.064791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.064260 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:31.104845 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.104323 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d-kube-api-access-5g286\" (UniqueName: \"kubernetes.io/projected/9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d-kube-api-access-5g286\") pod \"9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d\" (UID: \"9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d\") " Oct 30 20:59:31.104845 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.104542 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d-plugins-config-volume\") pod \"9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d\" (UID: \"9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d\") " Oct 30 20:59:31.105104 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.104867 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaaa8712-2a3c-42b8-897d-248d330bd675-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48\" (UID: \"eaaa8712-2a3c-42b8-897d-248d330bd675\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48" Oct 30 20:59:31.105177 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.105145 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv58c\" (UniqueName: \"kubernetes.io/projected/eaaa8712-2a3c-42b8-897d-248d330bd675-kube-api-access-lv58c\") pod \"payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48\" (UID: \"eaaa8712-2a3c-42b8-897d-248d330bd675\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48" Oct 30 20:59:31.145791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.134274 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d-kube-api-access-5g286" pod "9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d" (UID: "9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d"). InnerVolumeSpecName "kube-api-access-5g286". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:31.145791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.134377 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 20:59:31.145791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.134424 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" podUID="5c00d1f3-b911-40fd-8772-3ed4bba10d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 20:59:31.145791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.134482 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 20:59:31.146791 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.146163 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d-plugins-config-volume" pod "9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d" (UID: "9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:31.156643 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.156614 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn" Oct 30 20:59:31.160668 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.160641 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh" Oct 30 20:59:31.264161 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.264123 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:31.275944 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.275912 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv58c\" (UniqueName: \"kubernetes.io/projected/eaaa8712-2a3c-42b8-897d-248d330bd675-kube-api-access-lv58c\") pod \"payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48\" (UID: \"eaaa8712-2a3c-42b8-897d-248d330bd675\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48" Oct 30 20:59:31.279533 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.279501 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaaa8712-2a3c-42b8-897d-248d330bd675-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48\" (UID: \"eaaa8712-2a3c-42b8-897d-248d330bd675\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48" Oct 30 20:59:31.280075 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.280056 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:31.280215 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.280197 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g286\" (UniqueName: \"kubernetes.io/projected/9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d-kube-api-access-5g286\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:31.280475 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.280450 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaaa8712-2a3c-42b8-897d-248d330bd675-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48\" (UID: \"eaaa8712-2a3c-42b8-897d-248d330bd675\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48" Oct 30 20:59:31.391942 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.303343 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh" event={"ID":"9ac51789-e545-4331-ae29-f99e8150fc15","Type":"ContainerDied","Data":"78e7a0ad0a56ea8fe3c8faaa9e061ab51620ca9f3edede4de5dbaf754d276ca2"} Oct 30 20:59:31.392315 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.329469 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:31.400792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.342470 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv58c\" (UniqueName: \"kubernetes.io/projected/eaaa8712-2a3c-42b8-897d-248d330bd675-kube-api-access-lv58c\") pod \"payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48\" (UID: \"eaaa8712-2a3c-42b8-897d-248d330bd675\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48" Oct 30 20:59:31.400792 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.399566 4848 scope.go:122] "RemoveContainer" containerID="1b5b30e87296d1d849d232dd0af5765e48ddedd6bb10cbb2a12f30862aaff0c8" Oct 30 20:59:31.447709 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.433190 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz" Oct 30 20:59:31.454860 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.392276 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh"] Oct 30 20:59:31.458786 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.455043 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-c72dc9-66757c7c7f-drxrh"] Oct 30 20:59:31.462788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.458955 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz" event={"ID":"9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d","Type":"ContainerDied","Data":"fcc60ba37e0929bfd69f381c89c16452c6a8b81678e617298efeab36b5a55561"} Oct 30 20:59:31.482893 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.463925 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48" Oct 30 20:59:31.588920 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.588852 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz"] Oct 30 20:59:31.603044 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.602918 4848 scope.go:122] "RemoveContainer" containerID="c1cd4f243d2e2df8807d47c93c4b55f3c123bb7fe620cb135eee197f3f466745" Oct 30 20:59:31.642439 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.641943 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 20:59:31.642439 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.641999 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" podUID="1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 20:59:31.678754 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.678724 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-c72dc9-67b8c59867-hk4mz"] Oct 30 20:59:31.685273 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.684892 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:31.737788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.718856 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 20:59:31.737788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.718923 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" podUID="b0998438-c7f6-4cb0-ba25-b406d43b8959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 20:59:31.878873 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.862756 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m"] Oct 30 20:59:31.880261 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:31.880222 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" podUID="46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" containerName="istio-proxy" containerID="cri-o://d59ebc8b5ee90d2d693830b89d3c4a4bcf731592013ef870d811c31ab0773cca" gracePeriod=30 Oct 30 20:59:32.428608 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:32.285472 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:32.428608 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:32.285623 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:59:32.428608 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:32.330183 4848 patch_prober.go:29] interesting pod/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Oct 30 20:59:32.428608 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:32.336814 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" podUID="5c00d1f3-b911-40fd-8772-3ed4bba10d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Oct 30 20:59:32.665276 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:32.663200 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn"] Oct 30 20:59:32.718727 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:32.709275 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 20:59:32.718727 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:32.709333 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" podUID="b0998438-c7f6-4cb0-ba25-b406d43b8959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 20:59:32.829138 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:32.829097 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48"] Oct 30 20:59:32.863175 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:32.863146 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:59:32.865745 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:32.865725 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 20:59:33.002743 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:33.002704 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ac51789-e545-4331-ae29-f99e8150fc15" path="/var/lib/kubelet/pods/9ac51789-e545-4331-ae29-f99e8150fc15/volumes" Oct 30 20:59:33.004076 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:33.004049 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d" path="/var/lib/kubelet/pods/9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d/volumes" Oct 30 20:59:33.217761 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:33.217684 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:59:33.219806 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:33.219716 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 20:59:33.511622 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:33.511530 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn" event={"ID":"4e19f2d2-409b-457e-8f87-d4308b2c53ff","Type":"ContainerStarted","Data":"f70ca371d26e48325480616d746cb16e90bdbd2d9ed6c85c51839609e45b59df"} Oct 30 20:59:33.511622 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:33.511600 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn" event={"ID":"4e19f2d2-409b-457e-8f87-d4308b2c53ff","Type":"ContainerStarted","Data":"42d41711524a02fea7e938dddc7473548478d4b2285b2c068fd7851adad03cfc"} Oct 30 20:59:33.513302 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:33.513280 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn" Oct 30 20:59:33.515882 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:33.515858 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48" event={"ID":"eaaa8712-2a3c-42b8-897d-248d330bd675","Type":"ContainerStarted","Data":"62dacbaae253a9a1de122e5b1ae1ab354fdf5afdf55bc1051348097fb8166208"} Oct 30 20:59:33.516158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:33.516140 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48" Oct 30 20:59:33.516249 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:33.516184 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48" event={"ID":"eaaa8712-2a3c-42b8-897d-248d330bd675","Type":"ContainerStarted","Data":"97f8208e99c8a841e03f9c0772fb0b34bea887b994692472dfe4fec64b39304c"} Oct 30 20:59:33.532158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:33.532110 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn" podStartSLOduration=3.532094925 podStartE2EDuration="3.532094925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:33.531023212 +0000 UTC m=+1278.710919722" watchObservedRunningTime="2026-10-30 20:59:33.532094925 +0000 UTC m=+1278.711991434" Oct 30 20:59:33.550414 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:33.550362 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48" podStartSLOduration=3.5503507819999998 podStartE2EDuration="3.550350782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:33.548346942 +0000 UTC m=+1278.728243450" watchObservedRunningTime="2026-10-30 20:59:33.550350782 +0000 UTC m=+1278.730247309" Oct 30 20:59:33.734793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:33.733843 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:59:33.738801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:33.735044 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:59:33.738801 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:33.735634 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 20:59:34.433564 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:34.433538 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw" Oct 30 20:59:34.759732 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:34.759647 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb"] Oct 30 20:59:34.759983 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:34.759887 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" podUID="bed55d4a-5593-41db-84e1-fe96bb9a849b" containerName="istio-proxy" containerID="cri-o://54d4855122d3d655c940739ece767ff845b185cf404e5c2c43837e255d255622" gracePeriod=30 Oct 30 20:59:37.271113 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.271083 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:59:37.374796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.367867 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-envoy\") pod \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " Oct 30 20:59:37.374796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.367982 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-credential-socket\") pod \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " Oct 30 20:59:37.374796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.368025 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-podinfo\") pod \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " Oct 30 20:59:37.374796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.368125 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-workload-certs\") pod \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " Oct 30 20:59:37.374796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.368261 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-workload-socket\") pod \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " Oct 30 20:59:37.374796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.368344 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-data\") pod \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " Oct 30 20:59:37.374796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.368478 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-kube-api-access-8g7fb\" (UniqueName: \"kubernetes.io/projected/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-kube-api-access-8g7fb\") pod \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " Oct 30 20:59:37.374796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.368537 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istiod-ca-cert\") pod \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " Oct 30 20:59:37.374796 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.368582 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-token\" (UniqueName: \"kubernetes.io/projected/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-token\") pod \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\" (UID: \"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80\") " Oct 30 20:59:37.403851 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.400355 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-workload-socket" pod "46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" (UID: "46e85ca9-98d4-4a78-a5c6-a50d5d2cad80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:37.403851 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.400581 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-data" pod "46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" (UID: "46e85ca9-98d4-4a78-a5c6-a50d5d2cad80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:37.412007 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.410866 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-workload-certs" pod "46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" (UID: "46e85ca9-98d4-4a78-a5c6-a50d5d2cad80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:37.420609 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.416083 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istiod-ca-cert" pod "46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" (UID: "46e85ca9-98d4-4a78-a5c6-a50d5d2cad80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:37.420609 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.418023 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-kube-api-access-8g7fb" pod "46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" (UID: "46e85ca9-98d4-4a78-a5c6-a50d5d2cad80"). InnerVolumeSpecName "kube-api-access-8g7fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:37.420609 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.420466 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-credential-socket" pod "46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" (UID: "46e85ca9-98d4-4a78-a5c6-a50d5d2cad80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:37.431474 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.429999 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-token" pod "46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" (UID: "46e85ca9-98d4-4a78-a5c6-a50d5d2cad80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:37.432119 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.432087 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-envoy" pod "46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" (UID: "46e85ca9-98d4-4a78-a5c6-a50d5d2cad80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:37.433680 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.433654 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-podinfo" pod "46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" (UID: "46e85ca9-98d4-4a78-a5c6-a50d5d2cad80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:59:37.470341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.470307 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:37.470341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.470337 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:37.470341 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.470349 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g7fb\" (UniqueName: \"kubernetes.io/projected/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-kube-api-access-8g7fb\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:37.470559 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.470358 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:37.470559 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.470367 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:37.470559 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.470376 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:37.470559 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.470385 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:37.470559 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.470393 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:37.470559 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.470402 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:37.592898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.592819 4848 generic.go:356] "Generic (PLEG): container finished" podID="46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" containerID="d59ebc8b5ee90d2d693830b89d3c4a4bcf731592013ef870d811c31ab0773cca" exitCode=0 Oct 30 20:59:37.592898 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.592890 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" Oct 30 20:59:37.593075 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.592903 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" event={"ID":"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80","Type":"ContainerDied","Data":"d59ebc8b5ee90d2d693830b89d3c4a4bcf731592013ef870d811c31ab0773cca"} Oct 30 20:59:37.593075 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.592977 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m" event={"ID":"46e85ca9-98d4-4a78-a5c6-a50d5d2cad80","Type":"ContainerDied","Data":"1c72d2dcbc99e93b562755913429435d779da6a8629fbff6f165337dee68302f"} Oct 30 20:59:37.593075 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.592999 4848 scope.go:122] "RemoveContainer" containerID="d59ebc8b5ee90d2d693830b89d3c4a4bcf731592013ef870d811c31ab0773cca" Oct 30 20:59:37.604371 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.604352 4848 scope.go:122] "RemoveContainer" containerID="9fcb9d3e9c7a735ea417b94694a721a519405bdeb6be2be0ad24e54d188492f5" Oct 30 20:59:37.614000 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.613976 4848 scope.go:122] "RemoveContainer" containerID="d59ebc8b5ee90d2d693830b89d3c4a4bcf731592013ef870d811c31ab0773cca" Oct 30 20:59:37.614272 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:59:37.614250 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d59ebc8b5ee90d2d693830b89d3c4a4bcf731592013ef870d811c31ab0773cca\": container with ID starting with d59ebc8b5ee90d2d693830b89d3c4a4bcf731592013ef870d811c31ab0773cca not found: ID does not exist" containerID="d59ebc8b5ee90d2d693830b89d3c4a4bcf731592013ef870d811c31ab0773cca" Oct 30 20:59:37.614336 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.614279 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59ebc8b5ee90d2d693830b89d3c4a4bcf731592013ef870d811c31ab0773cca"} err="failed to get container status \"d59ebc8b5ee90d2d693830b89d3c4a4bcf731592013ef870d811c31ab0773cca\": rpc error: code = NotFound desc = could not find container \"d59ebc8b5ee90d2d693830b89d3c4a4bcf731592013ef870d811c31ab0773cca\": container with ID starting with d59ebc8b5ee90d2d693830b89d3c4a4bcf731592013ef870d811c31ab0773cca not found: ID does not exist" Oct 30 20:59:37.614336 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.614301 4848 scope.go:122] "RemoveContainer" containerID="9fcb9d3e9c7a735ea417b94694a721a519405bdeb6be2be0ad24e54d188492f5" Oct 30 20:59:37.614602 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:59:37.614582 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fcb9d3e9c7a735ea417b94694a721a519405bdeb6be2be0ad24e54d188492f5\": container with ID starting with 9fcb9d3e9c7a735ea417b94694a721a519405bdeb6be2be0ad24e54d188492f5 not found: ID does not exist" containerID="9fcb9d3e9c7a735ea417b94694a721a519405bdeb6be2be0ad24e54d188492f5" Oct 30 20:59:37.614655 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.614609 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fcb9d3e9c7a735ea417b94694a721a519405bdeb6be2be0ad24e54d188492f5"} err="failed to get container status \"9fcb9d3e9c7a735ea417b94694a721a519405bdeb6be2be0ad24e54d188492f5\": rpc error: code = NotFound desc = could not find container \"9fcb9d3e9c7a735ea417b94694a721a519405bdeb6be2be0ad24e54d188492f5\": container with ID starting with 9fcb9d3e9c7a735ea417b94694a721a519405bdeb6be2be0ad24e54d188492f5 not found: ID does not exist" Oct 30 20:59:37.617215 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.617196 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m"] Oct 30 20:59:37.621509 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:37.621491 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-2b7ca7-openshift-default-5cf454c79c-qfs6m"] Oct 30 20:59:38.007823 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.007789 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt"] Oct 30 20:59:38.008076 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.008046 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" podUID="c3b3cf05-d522-401f-81e2-3f221bc68b81" containerName="maas-api" containerID="cri-o://db7745141ad7c0ea45a9b0658d9a5c2478daa8c2347d60f83fea495cd8faaeae" gracePeriod=30 Oct 30 20:59:38.483430 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.483400 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" Oct 30 20:59:38.603393 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.603359 4848 generic.go:356] "Generic (PLEG): container finished" podID="c3b3cf05-d522-401f-81e2-3f221bc68b81" containerID="db7745141ad7c0ea45a9b0658d9a5c2478daa8c2347d60f83fea495cd8faaeae" exitCode=0 Oct 30 20:59:38.603583 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.603410 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" Oct 30 20:59:38.603583 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.603443 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" event={"ID":"c3b3cf05-d522-401f-81e2-3f221bc68b81","Type":"ContainerDied","Data":"db7745141ad7c0ea45a9b0658d9a5c2478daa8c2347d60f83fea495cd8faaeae"} Oct 30 20:59:38.603583 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.603505 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt" event={"ID":"c3b3cf05-d522-401f-81e2-3f221bc68b81","Type":"ContainerDied","Data":"1d32b12fbd83b4ed314a9b23395c99fd6b2f6098730fe1b8725139f67563fea7"} Oct 30 20:59:38.603583 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.603535 4848 scope.go:122] "RemoveContainer" containerID="db7745141ad7c0ea45a9b0658d9a5c2478daa8c2347d60f83fea495cd8faaeae" Oct 30 20:59:38.610287 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.610261 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3b3cf05-d522-401f-81e2-3f221bc68b81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b3cf05-d522-401f-81e2-3f221bc68b81-openshift-service-ca\") pod \"c3b3cf05-d522-401f-81e2-3f221bc68b81\" (UID: \"c3b3cf05-d522-401f-81e2-3f221bc68b81\") " Oct 30 20:59:38.610417 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.610391 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b3cf05-d522-401f-81e2-3f221bc68b81-kube-api-access-vvkzh\" (UniqueName: \"kubernetes.io/projected/c3b3cf05-d522-401f-81e2-3f221bc68b81-kube-api-access-vvkzh\") pod \"c3b3cf05-d522-401f-81e2-3f221bc68b81\" (UID: \"c3b3cf05-d522-401f-81e2-3f221bc68b81\") " Oct 30 20:59:38.610474 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.610430 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3b3cf05-d522-401f-81e2-3f221bc68b81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3b3cf05-d522-401f-81e2-3f221bc68b81-maas-api-tls\") pod \"c3b3cf05-d522-401f-81e2-3f221bc68b81\" (UID: \"c3b3cf05-d522-401f-81e2-3f221bc68b81\") " Oct 30 20:59:38.610610 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.610590 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3b3cf05-d522-401f-81e2-3f221bc68b81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3b3cf05-d522-401f-81e2-3f221bc68b81-metrics-certs\") pod \"c3b3cf05-d522-401f-81e2-3f221bc68b81\" (UID: \"c3b3cf05-d522-401f-81e2-3f221bc68b81\") " Oct 30 20:59:38.610700 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.610631 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b3cf05-d522-401f-81e2-3f221bc68b81-openshift-service-ca" pod "c3b3cf05-d522-401f-81e2-3f221bc68b81" (UID: "c3b3cf05-d522-401f-81e2-3f221bc68b81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:38.611237 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.611210 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3b3cf05-d522-401f-81e2-3f221bc68b81-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:38.613186 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.613118 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3b3cf05-d522-401f-81e2-3f221bc68b81-metrics-certs" pod "c3b3cf05-d522-401f-81e2-3f221bc68b81" (UID: "c3b3cf05-d522-401f-81e2-3f221bc68b81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:38.618729 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.618497 4848 scope.go:122] "RemoveContainer" containerID="db7745141ad7c0ea45a9b0658d9a5c2478daa8c2347d60f83fea495cd8faaeae" Oct 30 20:59:38.624478 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.622727 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3b3cf05-d522-401f-81e2-3f221bc68b81-maas-api-tls" pod "c3b3cf05-d522-401f-81e2-3f221bc68b81" (UID: "c3b3cf05-d522-401f-81e2-3f221bc68b81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:38.660788 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:59:38.638339 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7745141ad7c0ea45a9b0658d9a5c2478daa8c2347d60f83fea495cd8faaeae\": container with ID starting with db7745141ad7c0ea45a9b0658d9a5c2478daa8c2347d60f83fea495cd8faaeae not found: ID does not exist" containerID="db7745141ad7c0ea45a9b0658d9a5c2478daa8c2347d60f83fea495cd8faaeae" Oct 30 20:59:38.660788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.638385 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7745141ad7c0ea45a9b0658d9a5c2478daa8c2347d60f83fea495cd8faaeae"} err="failed to get container status \"db7745141ad7c0ea45a9b0658d9a5c2478daa8c2347d60f83fea495cd8faaeae\": rpc error: code = NotFound desc = could not find container \"db7745141ad7c0ea45a9b0658d9a5c2478daa8c2347d60f83fea495cd8faaeae\": container with ID starting with db7745141ad7c0ea45a9b0658d9a5c2478daa8c2347d60f83fea495cd8faaeae not found: ID does not exist" Oct 30 20:59:38.660788 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.646078 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b3cf05-d522-401f-81e2-3f221bc68b81-kube-api-access-vvkzh" pod "c3b3cf05-d522-401f-81e2-3f221bc68b81" (UID: "c3b3cf05-d522-401f-81e2-3f221bc68b81"). InnerVolumeSpecName "kube-api-access-vvkzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:38.713329 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.713245 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvkzh\" (UniqueName: \"kubernetes.io/projected/c3b3cf05-d522-401f-81e2-3f221bc68b81-kube-api-access-vvkzh\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:38.713329 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.713285 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3b3cf05-d522-401f-81e2-3f221bc68b81-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:38.713329 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.713301 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3b3cf05-d522-401f-81e2-3f221bc68b81-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:38.935382 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.935349 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" path="/var/lib/kubelet/pods/46e85ca9-98d4-4a78-a5c6-a50d5d2cad80/volumes" Oct 30 20:59:38.935694 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.935680 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt"] Oct 30 20:59:38.935733 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:38.935711 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c72dc9-86fbc6d6d-d9rbt"] Oct 30 20:59:40.055543 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.055514 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:59:40.139538 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.139508 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed55d4a-5593-41db-84e1-fe96bb9a849b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bed55d4a-5593-41db-84e1-fe96bb9a849b-istiod-ca-cert\") pod \"bed55d4a-5593-41db-84e1-fe96bb9a849b\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " Oct 30 20:59:40.139731 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.139559 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-workload-certs\") pod \"bed55d4a-5593-41db-84e1-fe96bb9a849b\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " Oct 30 20:59:40.139731 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.139623 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-data\") pod \"bed55d4a-5593-41db-84e1-fe96bb9a849b\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " Oct 30 20:59:40.139884 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.139800 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-token\" (UniqueName: \"kubernetes.io/projected/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-token\") pod \"bed55d4a-5593-41db-84e1-fe96bb9a849b\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " Oct 30 20:59:40.139884 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.139870 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-envoy\") pod \"bed55d4a-5593-41db-84e1-fe96bb9a849b\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " Oct 30 20:59:40.139995 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.139931 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-workload-socket\") pod \"bed55d4a-5593-41db-84e1-fe96bb9a849b\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " Oct 30 20:59:40.139995 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.139961 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-workload-certs" pod "bed55d4a-5593-41db-84e1-fe96bb9a849b" (UID: "bed55d4a-5593-41db-84e1-fe96bb9a849b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:40.139995 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.139981 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed55d4a-5593-41db-84e1-fe96bb9a849b-kube-api-access-9gs52\" (UniqueName: \"kubernetes.io/projected/bed55d4a-5593-41db-84e1-fe96bb9a849b-kube-api-access-9gs52\") pod \"bed55d4a-5593-41db-84e1-fe96bb9a849b\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " Oct 30 20:59:40.140143 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.139992 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-data" pod "bed55d4a-5593-41db-84e1-fe96bb9a849b" (UID: "bed55d4a-5593-41db-84e1-fe96bb9a849b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:40.140143 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.140017 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-podinfo\") pod \"bed55d4a-5593-41db-84e1-fe96bb9a849b\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " Oct 30 20:59:40.140143 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.140018 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed55d4a-5593-41db-84e1-fe96bb9a849b-istiod-ca-cert" pod "bed55d4a-5593-41db-84e1-fe96bb9a849b" (UID: "bed55d4a-5593-41db-84e1-fe96bb9a849b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:40.140143 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.140079 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-credential-socket\") pod \"bed55d4a-5593-41db-84e1-fe96bb9a849b\" (UID: \"bed55d4a-5593-41db-84e1-fe96bb9a849b\") " Oct 30 20:59:40.140344 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.140191 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-workload-socket" pod "bed55d4a-5593-41db-84e1-fe96bb9a849b" (UID: "bed55d4a-5593-41db-84e1-fe96bb9a849b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:40.140476 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.140449 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-credential-socket" pod "bed55d4a-5593-41db-84e1-fe96bb9a849b" (UID: "bed55d4a-5593-41db-84e1-fe96bb9a849b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:40.140714 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.140681 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.140714 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.140707 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.140915 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.140721 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bed55d4a-5593-41db-84e1-fe96bb9a849b-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.140915 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.140736 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.140915 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.140749 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.142210 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.142182 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-podinfo" pod "bed55d4a-5593-41db-84e1-fe96bb9a849b" (UID: "bed55d4a-5593-41db-84e1-fe96bb9a849b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:59:40.142400 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.142381 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed55d4a-5593-41db-84e1-fe96bb9a849b-kube-api-access-9gs52" pod "bed55d4a-5593-41db-84e1-fe96bb9a849b" (UID: "bed55d4a-5593-41db-84e1-fe96bb9a849b"). InnerVolumeSpecName "kube-api-access-9gs52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:40.142509 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.142491 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-envoy" pod "bed55d4a-5593-41db-84e1-fe96bb9a849b" (UID: "bed55d4a-5593-41db-84e1-fe96bb9a849b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:59:40.142567 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.142534 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-token" pod "bed55d4a-5593-41db-84e1-fe96bb9a849b" (UID: "bed55d4a-5593-41db-84e1-fe96bb9a849b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:40.241722 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.241685 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gs52\" (UniqueName: \"kubernetes.io/projected/bed55d4a-5593-41db-84e1-fe96bb9a849b-kube-api-access-9gs52\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.241722 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.241717 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.241936 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.241731 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.241936 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.241745 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bed55d4a-5593-41db-84e1-fe96bb9a849b-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:40.621516 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.621485 4848 generic.go:356] "Generic (PLEG): container finished" podID="bed55d4a-5593-41db-84e1-fe96bb9a849b" containerID="54d4855122d3d655c940739ece767ff845b185cf404e5c2c43837e255d255622" exitCode=0 Oct 30 20:59:40.621688 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.621544 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" Oct 30 20:59:40.621688 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.621566 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" event={"ID":"bed55d4a-5593-41db-84e1-fe96bb9a849b","Type":"ContainerDied","Data":"54d4855122d3d655c940739ece767ff845b185cf404e5c2c43837e255d255622"} Oct 30 20:59:40.621688 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.621658 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb" event={"ID":"bed55d4a-5593-41db-84e1-fe96bb9a849b","Type":"ContainerDied","Data":"ba75e919363db80a0f2577dc7238fa49511a558c155a18a8929f53d874c4925a"} Oct 30 20:59:40.621887 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.621691 4848 scope.go:122] "RemoveContainer" containerID="54d4855122d3d655c940739ece767ff845b185cf404e5c2c43837e255d255622" Oct 30 20:59:40.632599 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.632582 4848 scope.go:122] "RemoveContainer" containerID="d898fb74ee6342dd97b9e02eeb7b940d39ef5af38b64dc2b5c7fe62aaa4abdd6" Oct 30 20:59:40.642412 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.641978 4848 scope.go:122] "RemoveContainer" containerID="54d4855122d3d655c940739ece767ff845b185cf404e5c2c43837e255d255622" Oct 30 20:59:40.642523 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:59:40.642468 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d4855122d3d655c940739ece767ff845b185cf404e5c2c43837e255d255622\": container with ID starting with 54d4855122d3d655c940739ece767ff845b185cf404e5c2c43837e255d255622 not found: ID does not exist" containerID="54d4855122d3d655c940739ece767ff845b185cf404e5c2c43837e255d255622" Oct 30 20:59:40.642523 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.642494 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d4855122d3d655c940739ece767ff845b185cf404e5c2c43837e255d255622"} err="failed to get container status \"54d4855122d3d655c940739ece767ff845b185cf404e5c2c43837e255d255622\": rpc error: code = NotFound desc = could not find container \"54d4855122d3d655c940739ece767ff845b185cf404e5c2c43837e255d255622\": container with ID starting with 54d4855122d3d655c940739ece767ff845b185cf404e5c2c43837e255d255622 not found: ID does not exist" Oct 30 20:59:40.642523 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.642513 4848 scope.go:122] "RemoveContainer" containerID="d898fb74ee6342dd97b9e02eeb7b940d39ef5af38b64dc2b5c7fe62aaa4abdd6" Oct 30 20:59:40.642861 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:59:40.642810 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d898fb74ee6342dd97b9e02eeb7b940d39ef5af38b64dc2b5c7fe62aaa4abdd6\": container with ID starting with d898fb74ee6342dd97b9e02eeb7b940d39ef5af38b64dc2b5c7fe62aaa4abdd6 not found: ID does not exist" containerID="d898fb74ee6342dd97b9e02eeb7b940d39ef5af38b64dc2b5c7fe62aaa4abdd6" Oct 30 20:59:40.642861 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.642836 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d898fb74ee6342dd97b9e02eeb7b940d39ef5af38b64dc2b5c7fe62aaa4abdd6"} err="failed to get container status \"d898fb74ee6342dd97b9e02eeb7b940d39ef5af38b64dc2b5c7fe62aaa4abdd6\": rpc error: code = NotFound desc = could not find container \"d898fb74ee6342dd97b9e02eeb7b940d39ef5af38b64dc2b5c7fe62aaa4abdd6\": container with ID starting with d898fb74ee6342dd97b9e02eeb7b940d39ef5af38b64dc2b5c7fe62aaa4abdd6 not found: ID does not exist" Oct 30 20:59:40.646117 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.646066 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb"] Oct 30 20:59:40.650561 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.650542 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-38c5ca-openshift-default-7fcd849fdf-svdlb"] Oct 30 20:59:40.933617 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.933584 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bed55d4a-5593-41db-84e1-fe96bb9a849b" path="/var/lib/kubelet/pods/bed55d4a-5593-41db-84e1-fe96bb9a849b/volumes" Oct 30 20:59:40.934115 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:40.934099 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3b3cf05-d522-401f-81e2-3f221bc68b81" path="/var/lib/kubelet/pods/c3b3cf05-d522-401f-81e2-3f221bc68b81/volumes" Oct 30 20:59:42.541656 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:42.541619 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r"] Oct 30 20:59:42.556475 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:42.556448 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr"] Oct 30 20:59:44.525914 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:44.525827 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48" Oct 30 20:59:45.533183 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:45.533150 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn" Oct 30 20:59:47.542040 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:47.541992 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r" podUID="6cc891d1-c6b8-4650-bd1a-e14115e157bb" containerName="payload-processing" containerID="cri-o://d67dd5a5ce1abd8e2fe99a53f6010e3f91267eda37215b1a8c8a448866cf6f27" gracePeriod=25 Oct 30 20:59:47.557395 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:47.557359 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr" podUID="168fbd96-d5d2-43ea-b18e-26cd242b53cb" containerName="payload-pre-processing" containerID="cri-o://fba97c65d09fe9999762734a3f1900d27f7bf2d4f039c45d8a5d59950776d8b9" gracePeriod=25 Oct 30 20:59:47.780929 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:47.780897 4848 generic.go:356] "Generic (PLEG): container finished" podID="6cc891d1-c6b8-4650-bd1a-e14115e157bb" containerID="d67dd5a5ce1abd8e2fe99a53f6010e3f91267eda37215b1a8c8a448866cf6f27" exitCode=0 Oct 30 20:59:47.781051 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:47.781000 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r" event={"ID":"6cc891d1-c6b8-4650-bd1a-e14115e157bb","Type":"ContainerDied","Data":"d67dd5a5ce1abd8e2fe99a53f6010e3f91267eda37215b1a8c8a448866cf6f27"} Oct 30 20:59:47.784730 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:47.784699 4848 generic.go:356] "Generic (PLEG): container finished" podID="168fbd96-d5d2-43ea-b18e-26cd242b53cb" containerID="fba97c65d09fe9999762734a3f1900d27f7bf2d4f039c45d8a5d59950776d8b9" exitCode=0 Oct 30 20:59:47.784866 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:47.784761 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr" event={"ID":"168fbd96-d5d2-43ea-b18e-26cd242b53cb","Type":"ContainerDied","Data":"fba97c65d09fe9999762734a3f1900d27f7bf2d4f039c45d8a5d59950776d8b9"} Oct 30 20:59:47.986356 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:47.986325 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr" Oct 30 20:59:47.994823 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:47.993453 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r" Oct 30 20:59:48.163221 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:48.163140 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cc891d1-c6b8-4650-bd1a-e14115e157bb-kube-api-access-8x4gz\" (UniqueName: \"kubernetes.io/projected/6cc891d1-c6b8-4650-bd1a-e14115e157bb-kube-api-access-8x4gz\") pod \"6cc891d1-c6b8-4650-bd1a-e14115e157bb\" (UID: \"6cc891d1-c6b8-4650-bd1a-e14115e157bb\") " Oct 30 20:59:48.163378 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:48.163354 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/168fbd96-d5d2-43ea-b18e-26cd242b53cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/168fbd96-d5d2-43ea-b18e-26cd242b53cb-plugins-config-volume\") pod \"168fbd96-d5d2-43ea-b18e-26cd242b53cb\" (UID: \"168fbd96-d5d2-43ea-b18e-26cd242b53cb\") " Oct 30 20:59:48.163431 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:48.163415 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cc891d1-c6b8-4650-bd1a-e14115e157bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc891d1-c6b8-4650-bd1a-e14115e157bb-plugins-config-volume\") pod \"6cc891d1-c6b8-4650-bd1a-e14115e157bb\" (UID: \"6cc891d1-c6b8-4650-bd1a-e14115e157bb\") " Oct 30 20:59:48.163465 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:48.163455 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168fbd96-d5d2-43ea-b18e-26cd242b53cb-kube-api-access-h4j4t\" (UniqueName: \"kubernetes.io/projected/168fbd96-d5d2-43ea-b18e-26cd242b53cb-kube-api-access-h4j4t\") pod \"168fbd96-d5d2-43ea-b18e-26cd242b53cb\" (UID: \"168fbd96-d5d2-43ea-b18e-26cd242b53cb\") " Oct 30 20:59:48.163737 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:48.163698 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168fbd96-d5d2-43ea-b18e-26cd242b53cb-plugins-config-volume" pod "168fbd96-d5d2-43ea-b18e-26cd242b53cb" (UID: "168fbd96-d5d2-43ea-b18e-26cd242b53cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:48.163889 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:48.163751 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc891d1-c6b8-4650-bd1a-e14115e157bb-plugins-config-volume" pod "6cc891d1-c6b8-4650-bd1a-e14115e157bb" (UID: "6cc891d1-c6b8-4650-bd1a-e14115e157bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:48.164088 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:48.164068 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/168fbd96-d5d2-43ea-b18e-26cd242b53cb-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:48.164088 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:48.164089 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc891d1-c6b8-4650-bd1a-e14115e157bb-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:48.165404 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:48.165380 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc891d1-c6b8-4650-bd1a-e14115e157bb-kube-api-access-8x4gz" pod "6cc891d1-c6b8-4650-bd1a-e14115e157bb" (UID: "6cc891d1-c6b8-4650-bd1a-e14115e157bb"). InnerVolumeSpecName "kube-api-access-8x4gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:48.165604 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:48.165581 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168fbd96-d5d2-43ea-b18e-26cd242b53cb-kube-api-access-h4j4t" pod "168fbd96-d5d2-43ea-b18e-26cd242b53cb" (UID: "168fbd96-d5d2-43ea-b18e-26cd242b53cb"). InnerVolumeSpecName "kube-api-access-h4j4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:48.265198 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:48.265165 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4j4t\" (UniqueName: \"kubernetes.io/projected/168fbd96-d5d2-43ea-b18e-26cd242b53cb-kube-api-access-h4j4t\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:48.265198 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:48.265192 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x4gz\" (UniqueName: \"kubernetes.io/projected/6cc891d1-c6b8-4650-bd1a-e14115e157bb-kube-api-access-8x4gz\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:48.972636 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:48.972605 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r" Oct 30 20:59:48.973205 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:48.972595 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r" event={"ID":"6cc891d1-c6b8-4650-bd1a-e14115e157bb","Type":"ContainerDied","Data":"c1079141147045247d2653c6630a28576ef36b0069c002a85a35a465c9dfa5c8"} Oct 30 20:59:48.973205 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:48.972750 4848 scope.go:122] "RemoveContainer" containerID="d67dd5a5ce1abd8e2fe99a53f6010e3f91267eda37215b1a8c8a448866cf6f27" Oct 30 20:59:48.974878 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:48.974851 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr" event={"ID":"168fbd96-d5d2-43ea-b18e-26cd242b53cb","Type":"ContainerDied","Data":"b4fe09970f1f258c5da6c98c4602545c682b822cf159a4980a9b1a26f0822170"} Oct 30 20:59:48.974998 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:48.974962 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr" Oct 30 20:59:48.986072 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:48.986050 4848 scope.go:122] "RemoveContainer" containerID="fba97c65d09fe9999762734a3f1900d27f7bf2d4f039c45d8a5d59950776d8b9" Oct 30 20:59:49.033319 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:49.033274 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr"] Oct 30 20:59:49.036387 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:49.036361 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-70cf03-56c58948c-fclqr"] Oct 30 20:59:49.039810 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:49.039786 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r"] Oct 30 20:59:49.042569 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:49.042550 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-70cf03-76b94589f-sgf9r"] Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.432249 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6"] Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433219 4848 cpu_manager.go:414] "containerMap: removing container" podUID="46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" containerName="istio-proxy" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433240 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" containerName="istio-proxy" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433265 4848 cpu_manager.go:414] "containerMap: removing container" podUID="c3b3cf05-d522-401f-81e2-3f221bc68b81" containerName="maas-api" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433273 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b3cf05-d522-401f-81e2-3f221bc68b81" containerName="maas-api" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433284 4848 cpu_manager.go:414] "containerMap: removing container" podUID="46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" containerName="istio-proxy" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433293 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" containerName="istio-proxy" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433306 4848 cpu_manager.go:414] "containerMap: removing container" podUID="168fbd96-d5d2-43ea-b18e-26cd242b53cb" containerName="payload-pre-processing" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433313 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168fbd96-d5d2-43ea-b18e-26cd242b53cb" containerName="payload-pre-processing" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433335 4848 cpu_manager.go:414] "containerMap: removing container" podUID="bed55d4a-5593-41db-84e1-fe96bb9a849b" containerName="istio-proxy" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433343 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed55d4a-5593-41db-84e1-fe96bb9a849b" containerName="istio-proxy" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433361 4848 cpu_manager.go:414] "containerMap: removing container" podUID="bed55d4a-5593-41db-84e1-fe96bb9a849b" containerName="istio-proxy" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433368 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed55d4a-5593-41db-84e1-fe96bb9a849b" containerName="istio-proxy" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433378 4848 cpu_manager.go:414] "containerMap: removing container" podUID="9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d" containerName="payload-pre-processing" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433386 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d" containerName="payload-pre-processing" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433428 4848 cpu_manager.go:414] "containerMap: removing container" podUID="6cc891d1-c6b8-4650-bd1a-e14115e157bb" containerName="payload-processing" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433436 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cc891d1-c6b8-4650-bd1a-e14115e157bb" containerName="payload-processing" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433577 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b3cf05-d522-401f-81e2-3f221bc68b81" containerName="maas-api" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433589 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" containerName="istio-proxy" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433600 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="168fbd96-d5d2-43ea-b18e-26cd242b53cb" containerName="payload-pre-processing" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433611 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed55d4a-5593-41db-84e1-fe96bb9a849b" containerName="istio-proxy" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433621 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed55d4a-5593-41db-84e1-fe96bb9a849b" containerName="istio-proxy" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433630 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad8dfa4-5eed-4e2b-8566-70ca26e59e4d" containerName="payload-pre-processing" Oct 30 20:59:50.434268 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.433648 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cc891d1-c6b8-4650-bd1a-e14115e157bb" containerName="payload-processing" Oct 30 20:59:50.519412 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.519381 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6"] Oct 30 20:59:50.519725 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.519710 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.537596 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.536867 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-7c7768-openshift-default-dockercfg-5qw2h\"" Oct 30 20:59:50.650158 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.649339 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt"] Oct 30 20:59:50.653093 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.653058 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw"] Oct 30 20:59:50.746812 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.746674 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfhm7\" (UniqueName: \"kubernetes.io/projected/3c8ef112-9213-4f15-959c-63e09bb7faf5-kube-api-access-tfhm7\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.746979 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.746852 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-data\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.746979 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.746889 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-envoy\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.746979 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.746921 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8ef112-9213-4f15-959c-63e09bb7faf5-istiod-ca-cert\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.747109 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.747055 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-workload-socket\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.747145 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.747116 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-credential-socket\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.747196 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.747180 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-workload-certs\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.747232 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.747219 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-token\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.747336 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.747315 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-podinfo\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.849032 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.848995 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-envoy\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.849206 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.849050 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8ef112-9213-4f15-959c-63e09bb7faf5-istiod-ca-cert\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.849206 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.849188 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-workload-socket\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.849291 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.849247 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-credential-socket\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.849327 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.849312 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-workload-certs\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.849371 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.849356 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-token\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.849426 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.849415 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-podinfo\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.849571 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.849552 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfhm7\" (UniqueName: \"kubernetes.io/projected/3c8ef112-9213-4f15-959c-63e09bb7faf5-kube-api-access-tfhm7\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.849670 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.849595 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-credential-socket\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.849734 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.849711 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-data\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.849808 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.849728 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8ef112-9213-4f15-959c-63e09bb7faf5-istiod-ca-cert\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.850053 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.850031 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-data\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.850166 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.850143 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-workload-certs\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.850225 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.850208 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-workload-socket\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.852014 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.851991 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-envoy\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.852172 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.852154 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-podinfo\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.858430 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.858410 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-token\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.858602 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.858586 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfhm7\" (UniqueName: \"kubernetes.io/projected/3c8ef112-9213-4f15-959c-63e09bb7faf5-kube-api-access-tfhm7\") pod \"e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:50.938951 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.938912 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="168fbd96-d5d2-43ea-b18e-26cd242b53cb" path="/var/lib/kubelet/pods/168fbd96-d5d2-43ea-b18e-26cd242b53cb/volumes" Oct 30 20:59:50.939433 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:50.939408 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cc891d1-c6b8-4650-bd1a-e14115e157bb" path="/var/lib/kubelet/pods/6cc891d1-c6b8-4650-bd1a-e14115e157bb/volumes" Oct 30 20:59:51.040602 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:51.034629 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw"] Oct 30 20:59:51.196570 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:51.154038 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:51.247428 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:51.247393 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z"] Oct 30 20:59:51.248099 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:51.248054 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" podUID="f0770330-1ef1-43c9-978c-f6d994dd9dc4" containerName="maas-api" containerID="cri-o://fc4c4b7737490bd1a6fdb03ab939a98a6d6f63c45ae99bb284b5fa10232663f2" gracePeriod=30 Oct 30 20:59:51.739560 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:51.739512 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6"] Oct 30 20:59:51.753723 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:51.752515 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:51.753723 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:51.752610 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:51.753723 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:51.752661 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:51.983989 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:51.983921 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" Oct 30 20:59:52.051346 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.051310 4848 generic.go:356] "Generic (PLEG): container finished" podID="f0770330-1ef1-43c9-978c-f6d994dd9dc4" containerID="fc4c4b7737490bd1a6fdb03ab939a98a6d6f63c45ae99bb284b5fa10232663f2" exitCode=0 Oct 30 20:59:52.051668 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.051649 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" event={"ID":"f0770330-1ef1-43c9-978c-f6d994dd9dc4","Type":"ContainerDied","Data":"fc4c4b7737490bd1a6fdb03ab939a98a6d6f63c45ae99bb284b5fa10232663f2"} Oct 30 20:59:52.051817 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.051802 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" event={"ID":"f0770330-1ef1-43c9-978c-f6d994dd9dc4","Type":"ContainerDied","Data":"1c01d9047f82def9d11634c8c11b338cd1d2c3ff548da308256ef2c3ff8620a1"} Oct 30 20:59:52.051973 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.051961 4848 scope.go:122] "RemoveContainer" containerID="fc4c4b7737490bd1a6fdb03ab939a98a6d6f63c45ae99bb284b5fa10232663f2" Oct 30 20:59:52.052199 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.052186 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z" Oct 30 20:59:52.057372 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.057337 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" event={"ID":"3c8ef112-9213-4f15-959c-63e09bb7faf5","Type":"ContainerStarted","Data":"15b5d23359bf9635b589a8cd2493ceec0ed416cbd845253702a20a9c66ca9c59"} Oct 30 20:59:52.057505 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.057399 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" event={"ID":"3c8ef112-9213-4f15-959c-63e09bb7faf5","Type":"ContainerStarted","Data":"fd1e6b9092af2c97f8fcf04f2830cf0d5239a02dd3654bac73d5cdd784f5e473"} Oct 30 20:59:52.069362 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.069341 4848 scope.go:122] "RemoveContainer" containerID="fc4c4b7737490bd1a6fdb03ab939a98a6d6f63c45ae99bb284b5fa10232663f2" Oct 30 20:59:52.069940 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:59:52.069905 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4c4b7737490bd1a6fdb03ab939a98a6d6f63c45ae99bb284b5fa10232663f2\": container with ID starting with fc4c4b7737490bd1a6fdb03ab939a98a6d6f63c45ae99bb284b5fa10232663f2 not found: ID does not exist" containerID="fc4c4b7737490bd1a6fdb03ab939a98a6d6f63c45ae99bb284b5fa10232663f2" Oct 30 20:59:52.070064 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.069943 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4c4b7737490bd1a6fdb03ab939a98a6d6f63c45ae99bb284b5fa10232663f2"} err="failed to get container status \"fc4c4b7737490bd1a6fdb03ab939a98a6d6f63c45ae99bb284b5fa10232663f2\": rpc error: code = NotFound desc = could not find container \"fc4c4b7737490bd1a6fdb03ab939a98a6d6f63c45ae99bb284b5fa10232663f2\": container with ID starting with fc4c4b7737490bd1a6fdb03ab939a98a6d6f63c45ae99bb284b5fa10232663f2 not found: ID does not exist" Oct 30 20:59:52.088179 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.087725 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" podStartSLOduration=2.087710105 podStartE2EDuration="2.087710105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:52.084259242 +0000 UTC m=+1297.264155751" watchObservedRunningTime="2026-10-30 20:59:52.087710105 +0000 UTC m=+1297.267606612" Oct 30 20:59:52.155879 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.155847 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:52.156443 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.156420 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 20:59:52.156559 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.156489 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" podUID="3c8ef112-9213-4f15-959c-63e09bb7faf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 20:59:52.174704 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.174672 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0770330-1ef1-43c9-978c-f6d994dd9dc4-kube-api-access-dtjg7\" (UniqueName: \"kubernetes.io/projected/f0770330-1ef1-43c9-978c-f6d994dd9dc4-kube-api-access-dtjg7\") pod \"f0770330-1ef1-43c9-978c-f6d994dd9dc4\" (UID: \"f0770330-1ef1-43c9-978c-f6d994dd9dc4\") " Oct 30 20:59:52.174883 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.174761 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0770330-1ef1-43c9-978c-f6d994dd9dc4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0770330-1ef1-43c9-978c-f6d994dd9dc4-maas-api-tls\") pod \"f0770330-1ef1-43c9-978c-f6d994dd9dc4\" (UID: \"f0770330-1ef1-43c9-978c-f6d994dd9dc4\") " Oct 30 20:59:52.174883 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.174823 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0770330-1ef1-43c9-978c-f6d994dd9dc4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0770330-1ef1-43c9-978c-f6d994dd9dc4-metrics-certs\") pod \"f0770330-1ef1-43c9-978c-f6d994dd9dc4\" (UID: \"f0770330-1ef1-43c9-978c-f6d994dd9dc4\") " Oct 30 20:59:52.174883 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.174846 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0770330-1ef1-43c9-978c-f6d994dd9dc4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0770330-1ef1-43c9-978c-f6d994dd9dc4-openshift-service-ca\") pod \"f0770330-1ef1-43c9-978c-f6d994dd9dc4\" (UID: \"f0770330-1ef1-43c9-978c-f6d994dd9dc4\") " Oct 30 20:59:52.175356 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.175331 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0770330-1ef1-43c9-978c-f6d994dd9dc4-openshift-service-ca" pod "f0770330-1ef1-43c9-978c-f6d994dd9dc4" (UID: "f0770330-1ef1-43c9-978c-f6d994dd9dc4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:52.175532 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.175506 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0770330-1ef1-43c9-978c-f6d994dd9dc4-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:52.177414 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.177390 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0770330-1ef1-43c9-978c-f6d994dd9dc4-metrics-certs" pod "f0770330-1ef1-43c9-978c-f6d994dd9dc4" (UID: "f0770330-1ef1-43c9-978c-f6d994dd9dc4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:52.177507 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.177428 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0770330-1ef1-43c9-978c-f6d994dd9dc4-kube-api-access-dtjg7" pod "f0770330-1ef1-43c9-978c-f6d994dd9dc4" (UID: "f0770330-1ef1-43c9-978c-f6d994dd9dc4"). InnerVolumeSpecName "kube-api-access-dtjg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:52.177682 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.177660 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0770330-1ef1-43c9-978c-f6d994dd9dc4-maas-api-tls" pod "f0770330-1ef1-43c9-978c-f6d994dd9dc4" (UID: "f0770330-1ef1-43c9-978c-f6d994dd9dc4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:52.280858 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.280350 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtjg7\" (UniqueName: \"kubernetes.io/projected/f0770330-1ef1-43c9-978c-f6d994dd9dc4-kube-api-access-dtjg7\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:52.280858 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.280391 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0770330-1ef1-43c9-978c-f6d994dd9dc4-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:52.280858 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.280406 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0770330-1ef1-43c9-978c-f6d994dd9dc4-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:52.382957 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.382912 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z"] Oct 30 20:59:52.386410 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.386380 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-70cf03-5884f6986d-n755z"] Oct 30 20:59:52.511282 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:59:52.511245 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 20:59:52.700378 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.700321 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt"] Oct 30 20:59:52.701008 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:52.700975 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" podUID="2a94b236-339e-47ef-b5ab-49a6af417392" containerName="maas-api" containerID="cri-o://5285e7ca950944fa7788929da68d3c39bcd9367076b7054c98177f743bcc6f18" gracePeriod=30 Oct 30 20:59:53.111834 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:53.108847 4848 generic.go:356] "Generic (PLEG): container finished" podID="2a94b236-339e-47ef-b5ab-49a6af417392" containerID="5285e7ca950944fa7788929da68d3c39bcd9367076b7054c98177f743bcc6f18" exitCode=0 Oct 30 20:59:53.111834 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:53.109250 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0770330-1ef1-43c9-978c-f6d994dd9dc4" path="/var/lib/kubelet/pods/f0770330-1ef1-43c9-978c-f6d994dd9dc4/volumes" Oct 30 20:59:53.111834 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:53.109657 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" event={"ID":"2a94b236-339e-47ef-b5ab-49a6af417392","Type":"ContainerDied","Data":"5285e7ca950944fa7788929da68d3c39bcd9367076b7054c98177f743bcc6f18"} Oct 30 20:59:53.164077 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:53.164048 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:53.173816 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:53.173779 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" Oct 30 20:59:53.316502 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:53.316463 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a94b236-339e-47ef-b5ab-49a6af417392-kube-api-access-wrxkx\" (UniqueName: \"kubernetes.io/projected/2a94b236-339e-47ef-b5ab-49a6af417392-kube-api-access-wrxkx\") pod \"2a94b236-339e-47ef-b5ab-49a6af417392\" (UID: \"2a94b236-339e-47ef-b5ab-49a6af417392\") " Oct 30 20:59:53.316802 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:53.316666 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a94b236-339e-47ef-b5ab-49a6af417392-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a94b236-339e-47ef-b5ab-49a6af417392-metrics-certs\") pod \"2a94b236-339e-47ef-b5ab-49a6af417392\" (UID: \"2a94b236-339e-47ef-b5ab-49a6af417392\") " Oct 30 20:59:53.316802 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:53.316704 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a94b236-339e-47ef-b5ab-49a6af417392-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a94b236-339e-47ef-b5ab-49a6af417392-maas-api-tls\") pod \"2a94b236-339e-47ef-b5ab-49a6af417392\" (UID: \"2a94b236-339e-47ef-b5ab-49a6af417392\") " Oct 30 20:59:53.317006 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:53.316906 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a94b236-339e-47ef-b5ab-49a6af417392-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a94b236-339e-47ef-b5ab-49a6af417392-openshift-service-ca\") pod \"2a94b236-339e-47ef-b5ab-49a6af417392\" (UID: \"2a94b236-339e-47ef-b5ab-49a6af417392\") " Oct 30 20:59:53.321418 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:53.321346 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a94b236-339e-47ef-b5ab-49a6af417392-openshift-service-ca" pod "2a94b236-339e-47ef-b5ab-49a6af417392" (UID: "2a94b236-339e-47ef-b5ab-49a6af417392"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:53.327899 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:53.327839 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a94b236-339e-47ef-b5ab-49a6af417392-maas-api-tls" pod "2a94b236-339e-47ef-b5ab-49a6af417392" (UID: "2a94b236-339e-47ef-b5ab-49a6af417392"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:53.335140 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:53.335065 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a94b236-339e-47ef-b5ab-49a6af417392-kube-api-access-wrxkx" pod "2a94b236-339e-47ef-b5ab-49a6af417392" (UID: "2a94b236-339e-47ef-b5ab-49a6af417392"). InnerVolumeSpecName "kube-api-access-wrxkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:53.360984 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:53.360899 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a94b236-339e-47ef-b5ab-49a6af417392-metrics-certs" pod "2a94b236-339e-47ef-b5ab-49a6af417392" (UID: "2a94b236-339e-47ef-b5ab-49a6af417392"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:59:53.422487 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:53.422452 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a94b236-339e-47ef-b5ab-49a6af417392-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:53.422487 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:53.422484 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a94b236-339e-47ef-b5ab-49a6af417392-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:53.422487 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:53.422495 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a94b236-339e-47ef-b5ab-49a6af417392-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:53.422731 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:53.422505 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrxkx\" (UniqueName: \"kubernetes.io/projected/2a94b236-339e-47ef-b5ab-49a6af417392-kube-api-access-wrxkx\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:54.122533 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:54.121673 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" Oct 30 20:59:54.124136 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:54.123525 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt" event={"ID":"2a94b236-339e-47ef-b5ab-49a6af417392","Type":"ContainerDied","Data":"5812b8bf5747c18c331b36aeb1f4c52c0b41c9fe1b44f16160e0914c66d981cb"} Oct 30 20:59:54.124136 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:54.123596 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:54.124136 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:54.123656 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 20:59:54.124136 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:54.123688 4848 scope.go:122] "RemoveContainer" containerID="5285e7ca950944fa7788929da68d3c39bcd9367076b7054c98177f743bcc6f18" Oct 30 20:59:54.165229 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:54.165197 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt"] Oct 30 20:59:54.169960 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:54.169926 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-99b00f-f9696fbd5-xtxwt"] Oct 30 20:59:54.938592 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:54.937604 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a94b236-339e-47ef-b5ab-49a6af417392" path="/var/lib/kubelet/pods/2a94b236-339e-47ef-b5ab-49a6af417392/volumes" Oct 30 20:59:55.638380 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.638344 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v"] Oct 30 20:59:55.638974 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.638959 4848 cpu_manager.go:414] "containerMap: removing container" podUID="f0770330-1ef1-43c9-978c-f6d994dd9dc4" containerName="maas-api" Oct 30 20:59:55.639025 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.638975 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0770330-1ef1-43c9-978c-f6d994dd9dc4" containerName="maas-api" Oct 30 20:59:55.639025 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.639022 4848 cpu_manager.go:414] "containerMap: removing container" podUID="2a94b236-339e-47ef-b5ab-49a6af417392" containerName="maas-api" Oct 30 20:59:55.639096 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.639028 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a94b236-339e-47ef-b5ab-49a6af417392" containerName="maas-api" Oct 30 20:59:55.639130 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.639125 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a94b236-339e-47ef-b5ab-49a6af417392" containerName="maas-api" Oct 30 20:59:55.639163 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.639139 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0770330-1ef1-43c9-978c-f6d994dd9dc4" containerName="maas-api" Oct 30 20:59:55.639163 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.639146 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e85ca9-98d4-4a78-a5c6-a50d5d2cad80" containerName="istio-proxy" Oct 30 20:59:55.644681 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.644664 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v" Oct 30 20:59:55.650273 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.650168 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt" podUID="477e7c99-1a03-4ea9-b4e3-97b7488ef355" containerName="payload-processing" containerID="cri-o://64c9d95316a498b9ee8ed7d5101965d5e9b9068300e69d8fdad31b5d13f7fdb9" gracePeriod=25 Oct 30 20:59:55.650516 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.650482 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7c7768\"" Oct 30 20:59:55.654124 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.653963 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw" podUID="ce16f073-8c67-4080-aa8d-3882ac35fc6c" containerName="payload-pre-processing" containerID="cri-o://f716e06319b2232a0a3e0761ddb462e4524dd6ae5517acaad19488226bb7580d" gracePeriod=25 Oct 30 20:59:55.654686 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.654653 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v"] Oct 30 20:59:55.672856 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.671196 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54e85613-de9d-471e-b363-0eeb8c7972fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v\" (UID: \"54e85613-de9d-471e-b363-0eeb8c7972fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v" Oct 30 20:59:55.672856 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.671266 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt9vb\" (UniqueName: \"kubernetes.io/projected/54e85613-de9d-471e-b363-0eeb8c7972fe-kube-api-access-nt9vb\") pod \"payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v\" (UID: \"54e85613-de9d-471e-b363-0eeb8c7972fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v" Oct 30 20:59:55.696188 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.696160 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf"] Oct 30 20:59:55.701908 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.701863 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf" Oct 30 20:59:55.709298 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.709256 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf"] Oct 30 20:59:55.709814 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.709792 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7c7768-dockercfg-8sm47\"" Oct 30 20:59:55.774793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.772644 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54e85613-de9d-471e-b363-0eeb8c7972fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v\" (UID: \"54e85613-de9d-471e-b363-0eeb8c7972fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v" Oct 30 20:59:55.774793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.772747 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt9vb\" (UniqueName: \"kubernetes.io/projected/54e85613-de9d-471e-b363-0eeb8c7972fe-kube-api-access-nt9vb\") pod \"payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v\" (UID: \"54e85613-de9d-471e-b363-0eeb8c7972fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v" Oct 30 20:59:55.774793 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.774231 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54e85613-de9d-471e-b363-0eeb8c7972fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v\" (UID: \"54e85613-de9d-471e-b363-0eeb8c7972fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v" Oct 30 20:59:55.890260 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.890172 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d53735c-521f-45e0-9653-84d72efd49e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf\" (UID: \"8d53735c-521f-45e0-9653-84d72efd49e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf" Oct 30 20:59:55.890717 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.890691 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smp9f\" (UniqueName: \"kubernetes.io/projected/8d53735c-521f-45e0-9653-84d72efd49e2-kube-api-access-smp9f\") pod \"payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf\" (UID: \"8d53735c-521f-45e0-9653-84d72efd49e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf" Oct 30 20:59:55.894673 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.894647 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt9vb\" (UniqueName: \"kubernetes.io/projected/54e85613-de9d-471e-b363-0eeb8c7972fe-kube-api-access-nt9vb\") pod \"payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v\" (UID: \"54e85613-de9d-471e-b363-0eeb8c7972fe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v" Oct 30 20:59:55.962941 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.962625 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v" Oct 30 20:59:55.993844 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.992749 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d53735c-521f-45e0-9653-84d72efd49e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf\" (UID: \"8d53735c-521f-45e0-9653-84d72efd49e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf" Oct 30 20:59:55.993844 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.992963 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smp9f\" (UniqueName: \"kubernetes.io/projected/8d53735c-521f-45e0-9653-84d72efd49e2-kube-api-access-smp9f\") pod \"payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf\" (UID: \"8d53735c-521f-45e0-9653-84d72efd49e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf" Oct 30 20:59:55.994666 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:55.994437 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d53735c-521f-45e0-9653-84d72efd49e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf\" (UID: \"8d53735c-521f-45e0-9653-84d72efd49e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf" Oct 30 20:59:56.008550 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.008488 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smp9f\" (UniqueName: \"kubernetes.io/projected/8d53735c-521f-45e0-9653-84d72efd49e2-kube-api-access-smp9f\") pod \"payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf\" (UID: \"8d53735c-521f-45e0-9653-84d72efd49e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf" Oct 30 20:59:56.035544 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.035340 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw" podUID="8049a5ba-174e-4576-9adb-9c4eac65f0b5" containerName="payload-processing" containerID="cri-o://02c402196b36d3e2028997424efa29ac6206f812dd784d233156fb07f3717d0f" gracePeriod=25 Oct 30 20:59:56.143204 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.143050 4848 generic.go:356] "Generic (PLEG): container finished" podID="477e7c99-1a03-4ea9-b4e3-97b7488ef355" containerID="64c9d95316a498b9ee8ed7d5101965d5e9b9068300e69d8fdad31b5d13f7fdb9" exitCode=0 Oct 30 20:59:56.143204 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.143144 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt" event={"ID":"477e7c99-1a03-4ea9-b4e3-97b7488ef355","Type":"ContainerDied","Data":"64c9d95316a498b9ee8ed7d5101965d5e9b9068300e69d8fdad31b5d13f7fdb9"} Oct 30 20:59:56.158266 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.157027 4848 generic.go:356] "Generic (PLEG): container finished" podID="ce16f073-8c67-4080-aa8d-3882ac35fc6c" containerID="f716e06319b2232a0a3e0761ddb462e4524dd6ae5517acaad19488226bb7580d" exitCode=0 Oct 30 20:59:56.158266 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.158174 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw" event={"ID":"ce16f073-8c67-4080-aa8d-3882ac35fc6c","Type":"ContainerDied","Data":"f716e06319b2232a0a3e0761ddb462e4524dd6ae5517acaad19488226bb7580d"} Oct 30 20:59:56.168469 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.167916 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt" Oct 30 20:59:56.226337 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.225984 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw" Oct 30 20:59:56.289740 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.289539 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf" Oct 30 20:59:56.302042 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.302017 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/477e7c99-1a03-4ea9-b4e3-97b7488ef355-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477e7c99-1a03-4ea9-b4e3-97b7488ef355-plugins-config-volume\") pod \"477e7c99-1a03-4ea9-b4e3-97b7488ef355\" (UID: \"477e7c99-1a03-4ea9-b4e3-97b7488ef355\") " Oct 30 20:59:56.302188 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.302150 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/477e7c99-1a03-4ea9-b4e3-97b7488ef355-kube-api-access-9ww5l\" (UniqueName: \"kubernetes.io/projected/477e7c99-1a03-4ea9-b4e3-97b7488ef355-kube-api-access-9ww5l\") pod \"477e7c99-1a03-4ea9-b4e3-97b7488ef355\" (UID: \"477e7c99-1a03-4ea9-b4e3-97b7488ef355\") " Oct 30 20:59:56.302260 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.302205 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce16f073-8c67-4080-aa8d-3882ac35fc6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce16f073-8c67-4080-aa8d-3882ac35fc6c-plugins-config-volume\") pod \"ce16f073-8c67-4080-aa8d-3882ac35fc6c\" (UID: \"ce16f073-8c67-4080-aa8d-3882ac35fc6c\") " Oct 30 20:59:56.302327 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.302316 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce16f073-8c67-4080-aa8d-3882ac35fc6c-kube-api-access-pkg69\" (UniqueName: \"kubernetes.io/projected/ce16f073-8c67-4080-aa8d-3882ac35fc6c-kube-api-access-pkg69\") pod \"ce16f073-8c67-4080-aa8d-3882ac35fc6c\" (UID: \"ce16f073-8c67-4080-aa8d-3882ac35fc6c\") " Oct 30 20:59:56.302642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.302431 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/477e7c99-1a03-4ea9-b4e3-97b7488ef355-plugins-config-volume" pod "477e7c99-1a03-4ea9-b4e3-97b7488ef355" (UID: "477e7c99-1a03-4ea9-b4e3-97b7488ef355"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:56.304187 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.302904 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477e7c99-1a03-4ea9-b4e3-97b7488ef355-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:56.304187 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.304007 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce16f073-8c67-4080-aa8d-3882ac35fc6c-plugins-config-volume" pod "ce16f073-8c67-4080-aa8d-3882ac35fc6c" (UID: "ce16f073-8c67-4080-aa8d-3882ac35fc6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:56.312997 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.312216 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/477e7c99-1a03-4ea9-b4e3-97b7488ef355-kube-api-access-9ww5l" pod "477e7c99-1a03-4ea9-b4e3-97b7488ef355" (UID: "477e7c99-1a03-4ea9-b4e3-97b7488ef355"). InnerVolumeSpecName "kube-api-access-9ww5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:56.312997 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.312758 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce16f073-8c67-4080-aa8d-3882ac35fc6c-kube-api-access-pkg69" pod "ce16f073-8c67-4080-aa8d-3882ac35fc6c" (UID: "ce16f073-8c67-4080-aa8d-3882ac35fc6c"). InnerVolumeSpecName "kube-api-access-pkg69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:56.316893 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.316815 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v"] Oct 30 20:59:56.384455 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.384404 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9"] Oct 30 20:59:56.391200 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.391081 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw" Oct 30 20:59:56.398502 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.398074 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv"] Oct 30 20:59:56.404514 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.404340 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ww5l\" (UniqueName: \"kubernetes.io/projected/477e7c99-1a03-4ea9-b4e3-97b7488ef355-kube-api-access-9ww5l\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:56.404514 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.404364 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce16f073-8c67-4080-aa8d-3882ac35fc6c-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:56.404514 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.404378 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkg69\" (UniqueName: \"kubernetes.io/projected/ce16f073-8c67-4080-aa8d-3882ac35fc6c-kube-api-access-pkg69\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:56.506240 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.505142 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8049a5ba-174e-4576-9adb-9c4eac65f0b5-kube-api-access-644c5\" (UniqueName: \"kubernetes.io/projected/8049a5ba-174e-4576-9adb-9c4eac65f0b5-kube-api-access-644c5\") pod \"8049a5ba-174e-4576-9adb-9c4eac65f0b5\" (UID: \"8049a5ba-174e-4576-9adb-9c4eac65f0b5\") " Oct 30 20:59:56.506240 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.505327 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8049a5ba-174e-4576-9adb-9c4eac65f0b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8049a5ba-174e-4576-9adb-9c4eac65f0b5-plugins-config-volume\") pod \"8049a5ba-174e-4576-9adb-9c4eac65f0b5\" (UID: \"8049a5ba-174e-4576-9adb-9c4eac65f0b5\") " Oct 30 20:59:56.506745 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.506711 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8049a5ba-174e-4576-9adb-9c4eac65f0b5-plugins-config-volume" pod "8049a5ba-174e-4576-9adb-9c4eac65f0b5" (UID: "8049a5ba-174e-4576-9adb-9c4eac65f0b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:59:56.509418 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.509377 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8049a5ba-174e-4576-9adb-9c4eac65f0b5-kube-api-access-644c5" pod "8049a5ba-174e-4576-9adb-9c4eac65f0b5" (UID: "8049a5ba-174e-4576-9adb-9c4eac65f0b5"). InnerVolumeSpecName "kube-api-access-644c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:59:56.607242 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.607211 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-644c5\" (UniqueName: \"kubernetes.io/projected/8049a5ba-174e-4576-9adb-9c4eac65f0b5-kube-api-access-644c5\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:56.607242 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.607240 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8049a5ba-174e-4576-9adb-9c4eac65f0b5-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:59:56.614798 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:56.614746 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf"] Oct 30 20:59:56.615638 ip-10-0-130-52 kubenswrapper[4848]: W1030 20:59:56.615613 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d53735c_521f_45e0_9653_84d72efd49e2.slice/crio-bf2356b38213d915edbce556f716ab34506b622c42a79c942c6de3dca5b6b5f3 WatchSource:0}: Error finding container bf2356b38213d915edbce556f716ab34506b622c42a79c942c6de3dca5b6b5f3: Status 404 returned error can't find the container with id bf2356b38213d915edbce556f716ab34506b622c42a79c942c6de3dca5b6b5f3 Oct 30 20:59:57.167005 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.166961 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf" event={"ID":"8d53735c-521f-45e0-9653-84d72efd49e2","Type":"ContainerStarted","Data":"49d64ea267963f68ca859917eea640fd01b3eb5cf30134305c042fbbce828ed9"} Oct 30 20:59:57.167463 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.167039 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf" Oct 30 20:59:57.167463 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.167067 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf" event={"ID":"8d53735c-521f-45e0-9653-84d72efd49e2","Type":"ContainerStarted","Data":"bf2356b38213d915edbce556f716ab34506b622c42a79c942c6de3dca5b6b5f3"} Oct 30 20:59:57.168383 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.168360 4848 generic.go:356] "Generic (PLEG): container finished" podID="8049a5ba-174e-4576-9adb-9c4eac65f0b5" containerID="02c402196b36d3e2028997424efa29ac6206f812dd784d233156fb07f3717d0f" exitCode=0 Oct 30 20:59:57.168497 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.168385 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw" Oct 30 20:59:57.168497 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.168428 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw" event={"ID":"8049a5ba-174e-4576-9adb-9c4eac65f0b5","Type":"ContainerDied","Data":"02c402196b36d3e2028997424efa29ac6206f812dd784d233156fb07f3717d0f"} Oct 30 20:59:57.168497 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.168475 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw" event={"ID":"8049a5ba-174e-4576-9adb-9c4eac65f0b5","Type":"ContainerDied","Data":"fef0c7150441bbb5944a31b28635a1d78402a48a21c33491ed9c63283d336af8"} Oct 30 20:59:57.168642 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.168505 4848 scope.go:122] "RemoveContainer" containerID="02c402196b36d3e2028997424efa29ac6206f812dd784d233156fb07f3717d0f" Oct 30 20:59:57.170046 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.170026 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt" event={"ID":"477e7c99-1a03-4ea9-b4e3-97b7488ef355","Type":"ContainerDied","Data":"41ce7462c9a4839811a32b781f0b26266774c02f0bc9fed5450114330443f806"} Oct 30 20:59:57.170144 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.170032 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt" Oct 30 20:59:57.171685 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.171667 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw" Oct 30 20:59:57.171842 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.171708 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw" event={"ID":"ce16f073-8c67-4080-aa8d-3882ac35fc6c","Type":"ContainerDied","Data":"56fa7c02f53cff1512f0dc1eed8d2e8682fabaf57e05d76f3358fac34544c359"} Oct 30 20:59:57.173730 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.173711 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v" event={"ID":"54e85613-de9d-471e-b363-0eeb8c7972fe","Type":"ContainerStarted","Data":"3f50f3509d183e2604e8840370ca8ca3ce11ef494ca7fec4c9428ec159ef8081"} Oct 30 20:59:57.173870 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.173743 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v" event={"ID":"54e85613-de9d-471e-b363-0eeb8c7972fe","Type":"ContainerStarted","Data":"8011c0549e2a3b31bf19127c756189d89b6164559889d649f650d495ade91e75"} Oct 30 20:59:57.174085 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.174060 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v" Oct 30 20:59:57.181923 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.181905 4848 scope.go:122] "RemoveContainer" containerID="02c402196b36d3e2028997424efa29ac6206f812dd784d233156fb07f3717d0f" Oct 30 20:59:57.182191 ip-10-0-130-52 kubenswrapper[4848]: E1030 20:59:57.182174 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c402196b36d3e2028997424efa29ac6206f812dd784d233156fb07f3717d0f\": container with ID starting with 02c402196b36d3e2028997424efa29ac6206f812dd784d233156fb07f3717d0f not found: ID does not exist" containerID="02c402196b36d3e2028997424efa29ac6206f812dd784d233156fb07f3717d0f" Oct 30 20:59:57.182248 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.182197 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c402196b36d3e2028997424efa29ac6206f812dd784d233156fb07f3717d0f"} err="failed to get container status \"02c402196b36d3e2028997424efa29ac6206f812dd784d233156fb07f3717d0f\": rpc error: code = NotFound desc = could not find container \"02c402196b36d3e2028997424efa29ac6206f812dd784d233156fb07f3717d0f\": container with ID starting with 02c402196b36d3e2028997424efa29ac6206f812dd784d233156fb07f3717d0f not found: ID does not exist" Oct 30 20:59:57.182248 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.182212 4848 scope.go:122] "RemoveContainer" containerID="64c9d95316a498b9ee8ed7d5101965d5e9b9068300e69d8fdad31b5d13f7fdb9" Oct 30 20:59:57.185007 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.184960 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf" podStartSLOduration=2.184944165 podStartE2EDuration="2.184944165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:57.183010763 +0000 UTC m=+1302.362907273" watchObservedRunningTime="2026-10-30 20:59:57.184944165 +0000 UTC m=+1302.364840672" Oct 30 20:59:57.194334 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.194307 4848 scope.go:122] "RemoveContainer" containerID="f716e06319b2232a0a3e0761ddb462e4524dd6ae5517acaad19488226bb7580d" Oct 30 20:59:57.197715 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.197692 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw"] Oct 30 20:59:57.205417 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.205239 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b80638-6944d5b6bf-rqphw"] Oct 30 20:59:57.217562 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.217530 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt"] Oct 30 20:59:57.222804 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.222745 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-99b00f-8685b9b56f-9zlkt"] Oct 30 20:59:57.237490 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.237444 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v" podStartSLOduration=2.237428291 podStartE2EDuration="2.237428291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:57.235433543 +0000 UTC m=+1302.415330042" watchObservedRunningTime="2026-10-30 20:59:57.237428291 +0000 UTC m=+1302.417324797" Oct 30 20:59:57.250030 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.250005 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw"] Oct 30 20:59:57.254123 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:57.254102 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-99b00f-775b484799-qvhvw"] Oct 30 20:59:58.183402 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:58.183370 4848 generic.go:356] "Generic (PLEG): container finished" podID="e9b4f116-4af8-4321-887c-1df07ed0cd5d" containerID="cfa3bcfdab31604d7d29849a2da76015971a05de5efb266754a476a3b0e5fab7" exitCode=0 Oct 30 20:59:58.183925 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:58.183412 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" event={"ID":"e9b4f116-4af8-4321-887c-1df07ed0cd5d","Type":"ContainerDied","Data":"cfa3bcfdab31604d7d29849a2da76015971a05de5efb266754a476a3b0e5fab7"} Oct 30 20:59:58.183925 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:58.183893 4848 scope.go:122] "RemoveContainer" containerID="cfa3bcfdab31604d7d29849a2da76015971a05de5efb266754a476a3b0e5fab7" Oct 30 20:59:58.185471 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:58.185438 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:58.185602 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:58.185511 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:58.185602 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:58.185542 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 20:59:58.941449 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:58.941413 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="477e7c99-1a03-4ea9-b4e3-97b7488ef355" path="/var/lib/kubelet/pods/477e7c99-1a03-4ea9-b4e3-97b7488ef355/volumes" Oct 30 20:59:58.941897 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:58.941882 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8049a5ba-174e-4576-9adb-9c4eac65f0b5" path="/var/lib/kubelet/pods/8049a5ba-174e-4576-9adb-9c4eac65f0b5/volumes" Oct 30 20:59:58.942331 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:58.942309 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce16f073-8c67-4080-aa8d-3882ac35fc6c" path="/var/lib/kubelet/pods/ce16f073-8c67-4080-aa8d-3882ac35fc6c/volumes" Oct 30 20:59:59.196171 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:59.196087 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" event={"ID":"e9b4f116-4af8-4321-887c-1df07ed0cd5d","Type":"ContainerStarted","Data":"927cacb73efd93ec62c1d8c048842facfa010a9b8b1761539817a79bc2459e60"} Oct 30 20:59:59.581749 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:59.581661 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2"] Oct 30 20:59:59.581931 ip-10-0-130-52 kubenswrapper[4848]: I1030 20:59:59.581877 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" podUID="2eace1e0-9882-47ec-8c5d-927d8de6ffe6" containerName="istio-proxy" containerID="cri-o://7546b89e4af05979052c575f4e4dd98bc22e433460a3c9c3b2c5a8b8d592502a" gracePeriod=30 Oct 30 21:00:00.146426 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.146389 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5"] Oct 30 21:00:00.147138 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.147117 4848 cpu_manager.go:414] "containerMap: removing container" podUID="ce16f073-8c67-4080-aa8d-3882ac35fc6c" containerName="payload-pre-processing" Oct 30 21:00:00.147276 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.147139 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce16f073-8c67-4080-aa8d-3882ac35fc6c" containerName="payload-pre-processing" Oct 30 21:00:00.147276 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.147155 4848 cpu_manager.go:414] "containerMap: removing container" podUID="8049a5ba-174e-4576-9adb-9c4eac65f0b5" containerName="payload-processing" Oct 30 21:00:00.147276 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.147162 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8049a5ba-174e-4576-9adb-9c4eac65f0b5" containerName="payload-processing" Oct 30 21:00:00.147276 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.147218 4848 cpu_manager.go:414] "containerMap: removing container" podUID="477e7c99-1a03-4ea9-b4e3-97b7488ef355" containerName="payload-processing" Oct 30 21:00:00.147276 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.147226 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="477e7c99-1a03-4ea9-b4e3-97b7488ef355" containerName="payload-processing" Oct 30 21:00:00.147531 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.147377 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="477e7c99-1a03-4ea9-b4e3-97b7488ef355" containerName="payload-processing" Oct 30 21:00:00.147531 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.147397 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce16f073-8c67-4080-aa8d-3882ac35fc6c" containerName="payload-pre-processing" Oct 30 21:00:00.147531 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.147406 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="8049a5ba-174e-4576-9adb-9c4eac65f0b5" containerName="payload-processing" Oct 30 21:00:00.152317 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.152296 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5" Oct 30 21:00:00.158224 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.158203 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 21:00:00.159697 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.159361 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:00:00.159697 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.159408 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" podUID="e9b4f116-4af8-4321-887c-1df07ed0cd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 21:00:00.181837 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.180275 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5"] Oct 30 21:00:00.242909 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.242093 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\"" Oct 30 21:00:00.244881 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.244851 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw"] Oct 30 21:00:00.252959 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.252936 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw"] Oct 30 21:00:00.253095 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.253034 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" Oct 30 21:00:00.293582 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.293551 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5fcaf4f-96eb-4f50-8de3-b5755dad797f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889900-j5hb5\" (UID: \"d5fcaf4f-96eb-4f50-8de3-b5755dad797f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5" Oct 30 21:00:00.293934 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.293911 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqg6q\" (UniqueName: \"kubernetes.io/projected/d5fcaf4f-96eb-4f50-8de3-b5755dad797f-kube-api-access-pqg6q\") pod \"maas-api-key-cleanup-29889900-j5hb5\" (UID: \"d5fcaf4f-96eb-4f50-8de3-b5755dad797f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5" Oct 30 21:00:00.398084 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.397994 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5fcaf4f-96eb-4f50-8de3-b5755dad797f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889900-j5hb5\" (UID: \"d5fcaf4f-96eb-4f50-8de3-b5755dad797f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5" Oct 30 21:00:00.398084 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.398075 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ae86a2-a837-481f-8e1d-73d846addfbf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw\" (UID: \"16ae86a2-a837-481f-8e1d-73d846addfbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" Oct 30 21:00:00.398335 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.398175 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxdwm\" (UniqueName: \"kubernetes.io/projected/16ae86a2-a837-481f-8e1d-73d846addfbf-kube-api-access-fxdwm\") pod \"maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw\" (UID: \"16ae86a2-a837-481f-8e1d-73d846addfbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" Oct 30 21:00:00.398335 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.398316 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqg6q\" (UniqueName: \"kubernetes.io/projected/d5fcaf4f-96eb-4f50-8de3-b5755dad797f-kube-api-access-pqg6q\") pod \"maas-api-key-cleanup-29889900-j5hb5\" (UID: \"d5fcaf4f-96eb-4f50-8de3-b5755dad797f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5" Oct 30 21:00:00.398745 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.398725 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5fcaf4f-96eb-4f50-8de3-b5755dad797f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889900-j5hb5\" (UID: \"d5fcaf4f-96eb-4f50-8de3-b5755dad797f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5" Oct 30 21:00:00.430937 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.430907 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqg6q\" (UniqueName: \"kubernetes.io/projected/d5fcaf4f-96eb-4f50-8de3-b5755dad797f-kube-api-access-pqg6q\") pod \"maas-api-key-cleanup-29889900-j5hb5\" (UID: \"d5fcaf4f-96eb-4f50-8de3-b5755dad797f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5" Oct 30 21:00:00.451499 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.451469 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w"] Oct 30 21:00:00.455864 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.455849 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w" Oct 30 21:00:00.461523 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.461497 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b80638-dockercfg-gw2j4\"" Oct 30 21:00:00.461649 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.461578 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b80638\"" Oct 30 21:00:00.465747 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.465714 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w"] Oct 30 21:00:00.465953 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.465937 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5" Oct 30 21:00:00.596791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.574186 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ae86a2-a837-481f-8e1d-73d846addfbf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw\" (UID: \"16ae86a2-a837-481f-8e1d-73d846addfbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" Oct 30 21:00:00.596791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.574233 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxdwm\" (UniqueName: \"kubernetes.io/projected/16ae86a2-a837-481f-8e1d-73d846addfbf-kube-api-access-fxdwm\") pod \"maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw\" (UID: \"16ae86a2-a837-481f-8e1d-73d846addfbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" Oct 30 21:00:00.596791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.574277 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv6b6\" (UniqueName: \"kubernetes.io/projected/be1606dd-387e-4ab0-aa7f-7e12be2c3323-kube-api-access-xv6b6\") pod \"payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w\" (UID: \"be1606dd-387e-4ab0-aa7f-7e12be2c3323\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w" Oct 30 21:00:00.596791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.574582 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be1606dd-387e-4ab0-aa7f-7e12be2c3323-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w\" (UID: \"be1606dd-387e-4ab0-aa7f-7e12be2c3323\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w" Oct 30 21:00:00.596791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.578892 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ae86a2-a837-481f-8e1d-73d846addfbf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw\" (UID: \"16ae86a2-a837-481f-8e1d-73d846addfbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" Oct 30 21:00:00.634599 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.634560 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxdwm\" (UniqueName: \"kubernetes.io/projected/16ae86a2-a837-481f-8e1d-73d846addfbf-kube-api-access-fxdwm\") pod \"maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw\" (UID: \"16ae86a2-a837-481f-8e1d-73d846addfbf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" Oct 30 21:00:00.660847 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.659330 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm"] Oct 30 21:00:00.712821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.679816 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be1606dd-387e-4ab0-aa7f-7e12be2c3323-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w\" (UID: \"be1606dd-387e-4ab0-aa7f-7e12be2c3323\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w" Oct 30 21:00:00.712821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.679927 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv6b6\" (UniqueName: \"kubernetes.io/projected/be1606dd-387e-4ab0-aa7f-7e12be2c3323-kube-api-access-xv6b6\") pod \"payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w\" (UID: \"be1606dd-387e-4ab0-aa7f-7e12be2c3323\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w" Oct 30 21:00:00.712821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.681255 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be1606dd-387e-4ab0-aa7f-7e12be2c3323-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w\" (UID: \"be1606dd-387e-4ab0-aa7f-7e12be2c3323\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w" Oct 30 21:00:00.764442 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.764410 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv6b6\" (UniqueName: \"kubernetes.io/projected/be1606dd-387e-4ab0-aa7f-7e12be2c3323-kube-api-access-xv6b6\") pod \"payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w\" (UID: \"be1606dd-387e-4ab0-aa7f-7e12be2c3323\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w" Oct 30 21:00:00.772231 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.772189 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm"] Oct 30 21:00:00.772384 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.772317 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm" Oct 30 21:00:00.776150 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.776067 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w" Oct 30 21:00:00.881566 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.880696 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" Oct 30 21:00:00.887386 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.886249 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1246b25-a2e4-4021-b38e-64a255ba4a39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm\" (UID: \"e1246b25-a2e4-4021-b38e-64a255ba4a39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm" Oct 30 21:00:00.887386 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.886364 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr52n\" (UniqueName: \"kubernetes.io/projected/e1246b25-a2e4-4021-b38e-64a255ba4a39-kube-api-access-kr52n\") pod \"payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm\" (UID: \"e1246b25-a2e4-4021-b38e-64a255ba4a39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm" Oct 30 21:00:00.987905 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.987874 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr52n\" (UniqueName: \"kubernetes.io/projected/e1246b25-a2e4-4021-b38e-64a255ba4a39-kube-api-access-kr52n\") pod \"payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm\" (UID: \"e1246b25-a2e4-4021-b38e-64a255ba4a39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm" Oct 30 21:00:00.988094 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.988072 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1246b25-a2e4-4021-b38e-64a255ba4a39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm\" (UID: \"e1246b25-a2e4-4021-b38e-64a255ba4a39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm" Oct 30 21:00:00.989088 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:00.989059 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1246b25-a2e4-4021-b38e-64a255ba4a39-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm\" (UID: \"e1246b25-a2e4-4021-b38e-64a255ba4a39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm" Oct 30 21:00:01.003838 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:01.003804 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5"] Oct 30 21:00:01.038699 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:01.038491 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr52n\" (UniqueName: \"kubernetes.io/projected/e1246b25-a2e4-4021-b38e-64a255ba4a39-kube-api-access-kr52n\") pod \"payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm\" (UID: \"e1246b25-a2e4-4021-b38e-64a255ba4a39\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm" Oct 30 21:00:01.133794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:01.129626 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm" Oct 30 21:00:01.159504 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:01.159476 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:00:01.159726 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:01.159697 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" podUID="e9b4f116-4af8-4321-887c-1df07ed0cd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 21:00:01.178090 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:01.178061 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w"] Oct 30 21:00:01.359789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:01.353742 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw"] Oct 30 21:00:01.370999 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:01.370933 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5" event={"ID":"d5fcaf4f-96eb-4f50-8de3-b5755dad797f","Type":"ContainerStarted","Data":"35c036186a12bc0b7cc3c53307d1d74cabeda8f30b3b113b2509baee2b2c62ca"} Oct 30 21:00:01.385348 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:01.385317 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9" podUID="8588f639-d98d-48f8-8d35-fdb56ad83f6c" containerName="payload-processing" containerID="cri-o://dea9d021d2031e0cc50da912ff92012d0fa5326a88dfed1fec63acd1b8fc5716" gracePeriod=25 Oct 30 21:00:01.399110 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:01.398935 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv" podUID="8867aa05-25c2-49c6-bd30-0f8cc53c985e" containerName="payload-pre-processing" containerID="cri-o://2f26ed374f182c03dc5736d30202c6b7b9035c72162b28ca967c0583999b5f2d" gracePeriod=25 Oct 30 21:00:01.698503 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:01.698470 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm"] Oct 30 21:00:01.970693 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:01.970668 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9" Oct 30 21:00:02.053078 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.053043 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74"] Oct 30 21:00:02.053905 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.053591 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" podUID="b0998438-c7f6-4cb0-ba25-b406d43b8959" containerName="istio-proxy" containerID="cri-o://6cf00af8879dcc3d0bbd9a3d31607986495738ceaf18332628ccac0cf8af7f26" gracePeriod=30 Oct 30 21:00:02.083395 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.077721 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8588f639-d98d-48f8-8d35-fdb56ad83f6c-kube-api-access-rx4g6\" (UniqueName: \"kubernetes.io/projected/8588f639-d98d-48f8-8d35-fdb56ad83f6c-kube-api-access-rx4g6\") pod \"8588f639-d98d-48f8-8d35-fdb56ad83f6c\" (UID: \"8588f639-d98d-48f8-8d35-fdb56ad83f6c\") " Oct 30 21:00:02.083983 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.083958 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8588f639-d98d-48f8-8d35-fdb56ad83f6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8588f639-d98d-48f8-8d35-fdb56ad83f6c-plugins-config-volume\") pod \"8588f639-d98d-48f8-8d35-fdb56ad83f6c\" (UID: \"8588f639-d98d-48f8-8d35-fdb56ad83f6c\") " Oct 30 21:00:02.086391 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.086358 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8588f639-d98d-48f8-8d35-fdb56ad83f6c-plugins-config-volume" pod "8588f639-d98d-48f8-8d35-fdb56ad83f6c" (UID: "8588f639-d98d-48f8-8d35-fdb56ad83f6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:02.092259 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.092235 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv" Oct 30 21:00:02.094286 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.094262 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8588f639-d98d-48f8-8d35-fdb56ad83f6c-kube-api-access-rx4g6" pod "8588f639-d98d-48f8-8d35-fdb56ad83f6c" (UID: "8588f639-d98d-48f8-8d35-fdb56ad83f6c"). InnerVolumeSpecName "kube-api-access-rx4g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:02.214016 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.213978 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:00:02.218899 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.218850 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" podUID="e9b4f116-4af8-4321-887c-1df07ed0cd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 21:00:02.219458 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.217422 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8867aa05-25c2-49c6-bd30-0f8cc53c985e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8867aa05-25c2-49c6-bd30-0f8cc53c985e-plugins-config-volume\") pod \"8867aa05-25c2-49c6-bd30-0f8cc53c985e\" (UID: \"8867aa05-25c2-49c6-bd30-0f8cc53c985e\") " Oct 30 21:00:02.367123 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.367031 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8867aa05-25c2-49c6-bd30-0f8cc53c985e-kube-api-access-6hlw9\" (UniqueName: \"kubernetes.io/projected/8867aa05-25c2-49c6-bd30-0f8cc53c985e-kube-api-access-6hlw9\") pod \"8867aa05-25c2-49c6-bd30-0f8cc53c985e\" (UID: \"8867aa05-25c2-49c6-bd30-0f8cc53c985e\") " Oct 30 21:00:02.368254 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.368228 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx4g6\" (UniqueName: \"kubernetes.io/projected/8588f639-d98d-48f8-8d35-fdb56ad83f6c-kube-api-access-rx4g6\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:02.368389 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.368376 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8588f639-d98d-48f8-8d35-fdb56ad83f6c-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:02.368733 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.325954 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8867aa05-25c2-49c6-bd30-0f8cc53c985e-plugins-config-volume" pod "8867aa05-25c2-49c6-bd30-0f8cc53c985e" (UID: "8867aa05-25c2-49c6-bd30-0f8cc53c985e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:02.391985 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.391944 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8867aa05-25c2-49c6-bd30-0f8cc53c985e-kube-api-access-6hlw9" pod "8867aa05-25c2-49c6-bd30-0f8cc53c985e" (UID: "8867aa05-25c2-49c6-bd30-0f8cc53c985e"). InnerVolumeSpecName "kube-api-access-6hlw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:02.402559 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.402522 4848 generic.go:356] "Generic (PLEG): container finished" podID="d5fcaf4f-96eb-4f50-8de3-b5755dad797f" containerID="c90dcdb5456030d739b046d133881e1af15773d37347bc6515d232a9ad3d6af8" exitCode=60 Oct 30 21:00:02.402739 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.402701 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5" event={"ID":"d5fcaf4f-96eb-4f50-8de3-b5755dad797f","Type":"ContainerDied","Data":"c90dcdb5456030d739b046d133881e1af15773d37347bc6515d232a9ad3d6af8"} Oct 30 21:00:02.402983 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.402965 4848 scope.go:122] "RemoveContainer" containerID="c90dcdb5456030d739b046d133881e1af15773d37347bc6515d232a9ad3d6af8" Oct 30 21:00:02.405221 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.405196 4848 generic.go:356] "Generic (PLEG): container finished" podID="8588f639-d98d-48f8-8d35-fdb56ad83f6c" containerID="dea9d021d2031e0cc50da912ff92012d0fa5326a88dfed1fec63acd1b8fc5716" exitCode=0 Oct 30 21:00:02.405436 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.405416 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9" event={"ID":"8588f639-d98d-48f8-8d35-fdb56ad83f6c","Type":"ContainerDied","Data":"dea9d021d2031e0cc50da912ff92012d0fa5326a88dfed1fec63acd1b8fc5716"} Oct 30 21:00:02.405582 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.405565 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9" event={"ID":"8588f639-d98d-48f8-8d35-fdb56ad83f6c","Type":"ContainerDied","Data":"5715e58465aef5a3eaa5f2aa3b2dfa354b3c2a831c59a2d32f4efcb71779de67"} Oct 30 21:00:02.405710 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.405696 4848 scope.go:122] "RemoveContainer" containerID="dea9d021d2031e0cc50da912ff92012d0fa5326a88dfed1fec63acd1b8fc5716" Oct 30 21:00:02.405982 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.405967 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9" Oct 30 21:00:02.410417 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.410397 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm" event={"ID":"e1246b25-a2e4-4021-b38e-64a255ba4a39","Type":"ContainerStarted","Data":"c26ca51e9ae4c1b03b5074fdb226bbb4cf19430eb7e14c75c79c57094da8c7eb"} Oct 30 21:00:02.410562 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.410544 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm" event={"ID":"e1246b25-a2e4-4021-b38e-64a255ba4a39","Type":"ContainerStarted","Data":"195da83271242e95b11ba50e6a318b344a36dd82b1767edfd702c3020f2d1afb"} Oct 30 21:00:02.410948 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.410932 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm" Oct 30 21:00:02.412670 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.412651 4848 generic.go:356] "Generic (PLEG): container finished" podID="8867aa05-25c2-49c6-bd30-0f8cc53c985e" containerID="2f26ed374f182c03dc5736d30202c6b7b9035c72162b28ca967c0583999b5f2d" exitCode=0 Oct 30 21:00:02.412880 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.412863 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv" event={"ID":"8867aa05-25c2-49c6-bd30-0f8cc53c985e","Type":"ContainerDied","Data":"2f26ed374f182c03dc5736d30202c6b7b9035c72162b28ca967c0583999b5f2d"} Oct 30 21:00:02.413014 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.412999 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv" event={"ID":"8867aa05-25c2-49c6-bd30-0f8cc53c985e","Type":"ContainerDied","Data":"7d77facb4b89f46a1ffdeb6301f5c9042761060484f9900f35edc449a3953717"} Oct 30 21:00:02.413156 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.413143 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv" Oct 30 21:00:02.415318 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.415299 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" event={"ID":"16ae86a2-a837-481f-8e1d-73d846addfbf","Type":"ContainerStarted","Data":"982963d41113185b1fac1523b32c0b4f5c038f5696933c40e75f7f499aa06d72"} Oct 30 21:00:02.415470 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.415455 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" event={"ID":"16ae86a2-a837-481f-8e1d-73d846addfbf","Type":"ContainerStarted","Data":"6e27de1fb63e0070649087dc82d5c80327d863aa06f4d4a57468df5ed550ed77"} Oct 30 21:00:02.442386 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.441892 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w" event={"ID":"be1606dd-387e-4ab0-aa7f-7e12be2c3323","Type":"ContainerStarted","Data":"95931456f8261191a87a54d27e0aab3225c998165b5e33130796b3a65857e686"} Oct 30 21:00:02.442386 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.441965 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w" event={"ID":"be1606dd-387e-4ab0-aa7f-7e12be2c3323","Type":"ContainerStarted","Data":"d3c4a8d57e5e594f5dc3f1f7f82e1ae345d4a097a7f3196045c7ec00ea16352b"} Oct 30 21:00:02.442386 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.442335 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w" Oct 30 21:00:02.470303 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.470195 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8867aa05-25c2-49c6-bd30-0f8cc53c985e-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:02.470303 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.470223 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hlw9\" (UniqueName: \"kubernetes.io/projected/8867aa05-25c2-49c6-bd30-0f8cc53c985e-kube-api-access-6hlw9\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:02.474177 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.474159 4848 scope.go:122] "RemoveContainer" containerID="dea9d021d2031e0cc50da912ff92012d0fa5326a88dfed1fec63acd1b8fc5716" Oct 30 21:00:02.474581 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:00:02.474534 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dea9d021d2031e0cc50da912ff92012d0fa5326a88dfed1fec63acd1b8fc5716\": container with ID starting with dea9d021d2031e0cc50da912ff92012d0fa5326a88dfed1fec63acd1b8fc5716 not found: ID does not exist" containerID="dea9d021d2031e0cc50da912ff92012d0fa5326a88dfed1fec63acd1b8fc5716" Oct 30 21:00:02.474645 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.474591 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dea9d021d2031e0cc50da912ff92012d0fa5326a88dfed1fec63acd1b8fc5716"} err="failed to get container status \"dea9d021d2031e0cc50da912ff92012d0fa5326a88dfed1fec63acd1b8fc5716\": rpc error: code = NotFound desc = could not find container \"dea9d021d2031e0cc50da912ff92012d0fa5326a88dfed1fec63acd1b8fc5716\": container with ID starting with dea9d021d2031e0cc50da912ff92012d0fa5326a88dfed1fec63acd1b8fc5716 not found: ID does not exist" Oct 30 21:00:02.474645 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.474615 4848 scope.go:122] "RemoveContainer" containerID="2f26ed374f182c03dc5736d30202c6b7b9035c72162b28ca967c0583999b5f2d" Oct 30 21:00:02.534291 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.520144 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm" podStartSLOduration=2.520086841 podStartE2EDuration="2.520086841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:02.517284426 +0000 UTC m=+1307.697180934" watchObservedRunningTime="2026-10-30 21:00:02.520086841 +0000 UTC m=+1307.699983348" Oct 30 21:00:02.534291 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.529526 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" podStartSLOduration=2.529512122 podStartE2EDuration="2.529512122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:02.485923641 +0000 UTC m=+1307.665820149" watchObservedRunningTime="2026-10-30 21:00:02.529512122 +0000 UTC m=+1307.709408631" Oct 30 21:00:02.547793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.546855 4848 scope.go:122] "RemoveContainer" containerID="2f26ed374f182c03dc5736d30202c6b7b9035c72162b28ca967c0583999b5f2d" Oct 30 21:00:02.549900 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.549856 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w" podStartSLOduration=2.549842437 podStartE2EDuration="2.549842437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:02.549325997 +0000 UTC m=+1307.729222505" watchObservedRunningTime="2026-10-30 21:00:02.549842437 +0000 UTC m=+1307.729738944" Oct 30 21:00:02.552204 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:00:02.551897 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f26ed374f182c03dc5736d30202c6b7b9035c72162b28ca967c0583999b5f2d\": container with ID starting with 2f26ed374f182c03dc5736d30202c6b7b9035c72162b28ca967c0583999b5f2d not found: ID does not exist" containerID="2f26ed374f182c03dc5736d30202c6b7b9035c72162b28ca967c0583999b5f2d" Oct 30 21:00:02.552204 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.551949 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f26ed374f182c03dc5736d30202c6b7b9035c72162b28ca967c0583999b5f2d"} err="failed to get container status \"2f26ed374f182c03dc5736d30202c6b7b9035c72162b28ca967c0583999b5f2d\": rpc error: code = NotFound desc = could not find container \"2f26ed374f182c03dc5736d30202c6b7b9035c72162b28ca967c0583999b5f2d\": container with ID starting with 2f26ed374f182c03dc5736d30202c6b7b9035c72162b28ca967c0583999b5f2d not found: ID does not exist" Oct 30 21:00:02.603215 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.603188 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv"] Oct 30 21:00:02.611797 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.610238 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-6c1acd-5f95894d95-drkcv"] Oct 30 21:00:02.642258 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.642166 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9"] Oct 30 21:00:02.645618 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.645595 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-6c1acd-5fc758cf89-rz9q9"] Oct 30 21:00:02.945372 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.945342 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8588f639-d98d-48f8-8d35-fdb56ad83f6c" path="/var/lib/kubelet/pods/8588f639-d98d-48f8-8d35-fdb56ad83f6c/volumes" Oct 30 21:00:02.946129 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:02.946110 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8867aa05-25c2-49c6-bd30-0f8cc53c985e" path="/var/lib/kubelet/pods/8867aa05-25c2-49c6-bd30-0f8cc53c985e/volumes" Oct 30 21:00:03.158621 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.158589 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:00:03.158824 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.158642 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" podUID="e9b4f116-4af8-4321-887c-1df07ed0cd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 21:00:03.471077 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.471034 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5" event={"ID":"d5fcaf4f-96eb-4f50-8de3-b5755dad797f","Type":"ContainerStarted","Data":"dfa4238d98b07b69a38a235050f23db4f9f46afb9acd646157f412750c7f94d0"} Oct 30 21:00:03.474822 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.474787 4848 generic.go:356] "Generic (PLEG): container finished" podID="16ae86a2-a837-481f-8e1d-73d846addfbf" containerID="982963d41113185b1fac1523b32c0b4f5c038f5696933c40e75f7f499aa06d72" exitCode=60 Oct 30 21:00:03.475358 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.475333 4848 scope.go:122] "RemoveContainer" containerID="982963d41113185b1fac1523b32c0b4f5c038f5696933c40e75f7f499aa06d72" Oct 30 21:00:03.476837 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.474830 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" event={"ID":"16ae86a2-a837-481f-8e1d-73d846addfbf","Type":"ContainerDied","Data":"982963d41113185b1fac1523b32c0b4f5c038f5696933c40e75f7f499aa06d72"} Oct 30 21:00:03.493791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.493632 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5" podStartSLOduration=3.493617095 podStartE2EDuration="3.493617095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:03.490727201 +0000 UTC m=+1308.670623708" watchObservedRunningTime="2026-10-30 21:00:03.493617095 +0000 UTC m=+1308.673513601" Oct 30 21:00:03.546053 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.546026 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h"] Oct 30 21:00:03.546622 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.546609 4848 cpu_manager.go:414] "containerMap: removing container" podUID="8588f639-d98d-48f8-8d35-fdb56ad83f6c" containerName="payload-processing" Oct 30 21:00:03.546677 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.546623 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8588f639-d98d-48f8-8d35-fdb56ad83f6c" containerName="payload-processing" Oct 30 21:00:03.546677 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.546650 4848 cpu_manager.go:414] "containerMap: removing container" podUID="8867aa05-25c2-49c6-bd30-0f8cc53c985e" containerName="payload-pre-processing" Oct 30 21:00:03.546677 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.546655 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8867aa05-25c2-49c6-bd30-0f8cc53c985e" containerName="payload-pre-processing" Oct 30 21:00:03.546811 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.546760 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="8588f639-d98d-48f8-8d35-fdb56ad83f6c" containerName="payload-processing" Oct 30 21:00:03.546811 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.546788 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="8867aa05-25c2-49c6-bd30-0f8cc53c985e" containerName="payload-pre-processing" Oct 30 21:00:03.551471 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.551453 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h" Oct 30 21:00:03.555597 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.555575 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:00:03.558778 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.558746 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h"] Oct 30 21:00:03.693753 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.693720 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe65968e-pt79h\" (UID: \"2cbd5d2a-36e7-4b27-afb0-a94a1465278b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h" Oct 30 21:00:03.693753 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.693757 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7nfp\" (UniqueName: \"kubernetes.io/projected/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-kube-api-access-x7nfp\") pod \"maas-api-revoke-sub-fe65968e-pt79h\" (UID: \"2cbd5d2a-36e7-4b27-afb0-a94a1465278b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h" Oct 30 21:00:03.693975 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.693947 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe65968e-pt79h\" (UID: \"2cbd5d2a-36e7-4b27-afb0-a94a1465278b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h" Oct 30 21:00:03.737377 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.737300 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 21:00:03.737377 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.737355 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" podUID="b0998438-c7f6-4cb0-ba25-b406d43b8959" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:00:03.794793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.794739 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe65968e-pt79h\" (UID: \"2cbd5d2a-36e7-4b27-afb0-a94a1465278b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h" Oct 30 21:00:03.794940 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.794801 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7nfp\" (UniqueName: \"kubernetes.io/projected/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-kube-api-access-x7nfp\") pod \"maas-api-revoke-sub-fe65968e-pt79h\" (UID: \"2cbd5d2a-36e7-4b27-afb0-a94a1465278b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h" Oct 30 21:00:03.795001 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.794983 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe65968e-pt79h\" (UID: \"2cbd5d2a-36e7-4b27-afb0-a94a1465278b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h" Oct 30 21:00:03.795426 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.795408 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe65968e-pt79h\" (UID: \"2cbd5d2a-36e7-4b27-afb0-a94a1465278b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h" Oct 30 21:00:03.797346 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.797322 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe65968e-pt79h\" (UID: \"2cbd5d2a-36e7-4b27-afb0-a94a1465278b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h" Oct 30 21:00:03.824468 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.824436 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7nfp\" (UniqueName: \"kubernetes.io/projected/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-kube-api-access-x7nfp\") pod \"maas-api-revoke-sub-fe65968e-pt79h\" (UID: \"2cbd5d2a-36e7-4b27-afb0-a94a1465278b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h" Oct 30 21:00:03.946303 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:03.946278 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h" Oct 30 21:00:04.158445 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.158405 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 30 21:00:04.158623 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.158458 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" podUID="e9b4f116-4af8-4321-887c-1df07ed0cd5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 30 21:00:04.237308 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.237281 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h"] Oct 30 21:00:04.238998 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:00:04.238971 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cbd5d2a_36e7_4b27_afb0_a94a1465278b.slice/crio-da27ef042ea2c8ad398c7e8b7acd8e684a8ebff504e5005092da9615a3f76f78 WatchSource:0}: Error finding container da27ef042ea2c8ad398c7e8b7acd8e684a8ebff504e5005092da9615a3f76f78: Status 404 returned error can't find the container with id da27ef042ea2c8ad398c7e8b7acd8e684a8ebff504e5005092da9615a3f76f78 Oct 30 21:00:04.297178 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.297145 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w"] Oct 30 21:00:04.302836 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.302811 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" Oct 30 21:00:04.341662 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.341621 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w"] Oct 30 21:00:04.351782 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.351610 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-7c7768-serving-cert\"" Oct 30 21:00:04.377811 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.377665 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 21:00:04.403053 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.403012 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xll9v\" (UniqueName: \"kubernetes.io/projected/75e137a8-1096-4138-9e30-2cce04668543-kube-api-access-xll9v\") pod \"maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w\" (UID: \"75e137a8-1096-4138-9e30-2cce04668543\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" Oct 30 21:00:04.403235 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.403139 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e137a8-1096-4138-9e30-2cce04668543-metrics-certs\") pod \"maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w\" (UID: \"75e137a8-1096-4138-9e30-2cce04668543\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" Oct 30 21:00:04.403235 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.403176 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e137a8-1096-4138-9e30-2cce04668543-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w\" (UID: \"75e137a8-1096-4138-9e30-2cce04668543\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" Oct 30 21:00:04.403385 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.403349 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e137a8-1096-4138-9e30-2cce04668543-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w\" (UID: \"75e137a8-1096-4138-9e30-2cce04668543\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" Oct 30 21:00:04.494417 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.494384 4848 generic.go:356] "Generic (PLEG): container finished" podID="d5fcaf4f-96eb-4f50-8de3-b5755dad797f" containerID="dfa4238d98b07b69a38a235050f23db4f9f46afb9acd646157f412750c7f94d0" exitCode=60 Oct 30 21:00:04.494916 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.494464 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5" event={"ID":"d5fcaf4f-96eb-4f50-8de3-b5755dad797f","Type":"ContainerDied","Data":"dfa4238d98b07b69a38a235050f23db4f9f46afb9acd646157f412750c7f94d0"} Oct 30 21:00:04.494916 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.494819 4848 scope.go:122] "RemoveContainer" containerID="dfa4238d98b07b69a38a235050f23db4f9f46afb9acd646157f412750c7f94d0" Oct 30 21:00:04.495062 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.494929 4848 scope.go:122] "RemoveContainer" containerID="c90dcdb5456030d739b046d133881e1af15773d37347bc6515d232a9ad3d6af8" Oct 30 21:00:04.495149 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:00:04.495126 4848 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889900-j5hb5_odh-ai-gateway-infra(d5fcaf4f-96eb-4f50-8de3-b5755dad797f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5" podUID="d5fcaf4f-96eb-4f50-8de3-b5755dad797f" Oct 30 21:00:04.496339 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.496315 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h" event={"ID":"2cbd5d2a-36e7-4b27-afb0-a94a1465278b","Type":"ContainerStarted","Data":"da27ef042ea2c8ad398c7e8b7acd8e684a8ebff504e5005092da9615a3f76f78"} Oct 30 21:00:04.498548 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.498522 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" event={"ID":"16ae86a2-a837-481f-8e1d-73d846addfbf","Type":"ContainerStarted","Data":"9576b230d5aa2c41c02bfdd0a047353c50644a77ef4254c7c22d0df0b09ef78e"} Oct 30 21:00:04.534866 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.526797 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e137a8-1096-4138-9e30-2cce04668543-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w\" (UID: \"75e137a8-1096-4138-9e30-2cce04668543\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" Oct 30 21:00:04.534866 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.526924 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xll9v\" (UniqueName: \"kubernetes.io/projected/75e137a8-1096-4138-9e30-2cce04668543-kube-api-access-xll9v\") pod \"maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w\" (UID: \"75e137a8-1096-4138-9e30-2cce04668543\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" Oct 30 21:00:04.534866 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.527044 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e137a8-1096-4138-9e30-2cce04668543-metrics-certs\") pod \"maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w\" (UID: \"75e137a8-1096-4138-9e30-2cce04668543\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" Oct 30 21:00:04.534866 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.527076 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e137a8-1096-4138-9e30-2cce04668543-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w\" (UID: \"75e137a8-1096-4138-9e30-2cce04668543\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" Oct 30 21:00:04.572955 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.570423 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e137a8-1096-4138-9e30-2cce04668543-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w\" (UID: \"75e137a8-1096-4138-9e30-2cce04668543\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" Oct 30 21:00:04.613865 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.608279 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xll9v\" (UniqueName: \"kubernetes.io/projected/75e137a8-1096-4138-9e30-2cce04668543-kube-api-access-xll9v\") pod \"maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w\" (UID: \"75e137a8-1096-4138-9e30-2cce04668543\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" Oct 30 21:00:04.613865 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.608881 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e137a8-1096-4138-9e30-2cce04668543-metrics-certs\") pod \"maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w\" (UID: \"75e137a8-1096-4138-9e30-2cce04668543\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" Oct 30 21:00:04.613865 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.610928 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e137a8-1096-4138-9e30-2cce04668543-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w\" (UID: \"75e137a8-1096-4138-9e30-2cce04668543\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" Oct 30 21:00:04.663943 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.663920 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" Oct 30 21:00:04.862199 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.861864 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 21:00:04.936566 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.935010 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-kube-api-access-d7dh7\" (UniqueName: \"kubernetes.io/projected/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-kube-api-access-d7dh7\") pod \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " Oct 30 21:00:04.936566 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.935125 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-credential-socket\") pod \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " Oct 30 21:00:04.936566 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.935176 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-workload-socket\") pod \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " Oct 30 21:00:04.936566 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.935249 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-podinfo\") pod \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " Oct 30 21:00:04.936566 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.935286 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-token\" (UniqueName: \"kubernetes.io/projected/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-token\") pod \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " Oct 30 21:00:04.936566 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.935375 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-data\") pod \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " Oct 30 21:00:04.936566 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.935402 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-workload-certs\") pod \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " Oct 30 21:00:04.936566 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.935443 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istiod-ca-cert\") pod \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " Oct 30 21:00:04.936566 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.936348 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istiod-ca-cert" pod "2eace1e0-9882-47ec-8c5d-927d8de6ffe6" (UID: "2eace1e0-9882-47ec-8c5d-927d8de6ffe6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:04.937340 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.937029 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-data" pod "2eace1e0-9882-47ec-8c5d-927d8de6ffe6" (UID: "2eace1e0-9882-47ec-8c5d-927d8de6ffe6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:04.937340 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.937219 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-workload-certs" pod "2eace1e0-9882-47ec-8c5d-927d8de6ffe6" (UID: "2eace1e0-9882-47ec-8c5d-927d8de6ffe6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:04.938376 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.937394 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-credential-socket" pod "2eace1e0-9882-47ec-8c5d-927d8de6ffe6" (UID: "2eace1e0-9882-47ec-8c5d-927d8de6ffe6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:04.938563 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.938442 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-workload-socket" pod "2eace1e0-9882-47ec-8c5d-927d8de6ffe6" (UID: "2eace1e0-9882-47ec-8c5d-927d8de6ffe6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:04.938894 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.938832 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-podinfo" pod "2eace1e0-9882-47ec-8c5d-927d8de6ffe6" (UID: "2eace1e0-9882-47ec-8c5d-927d8de6ffe6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:00:04.946755 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.946726 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-kube-api-access-d7dh7" pod "2eace1e0-9882-47ec-8c5d-927d8de6ffe6" (UID: "2eace1e0-9882-47ec-8c5d-927d8de6ffe6"). InnerVolumeSpecName "kube-api-access-d7dh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:04.947207 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.947187 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-token" pod "2eace1e0-9882-47ec-8c5d-927d8de6ffe6" (UID: "2eace1e0-9882-47ec-8c5d-927d8de6ffe6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:04.973090 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:04.973010 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w"] Oct 30 21:00:04.978152 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:00:04.978117 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e137a8_1096_4138_9e30_2cce04668543.slice/crio-7bc275d243e35737d56010fc880bd75497c92d88ebf0b23f627a04af15cf1dfa WatchSource:0}: Error finding container 7bc275d243e35737d56010fc880bd75497c92d88ebf0b23f627a04af15cf1dfa: Status 404 returned error can't find the container with id 7bc275d243e35737d56010fc880bd75497c92d88ebf0b23f627a04af15cf1dfa Oct 30 21:00:05.041957 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.041086 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-envoy\") pod \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\" (UID: \"2eace1e0-9882-47ec-8c5d-927d8de6ffe6\") " Oct 30 21:00:05.041957 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.041742 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7dh7\" (UniqueName: \"kubernetes.io/projected/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-kube-api-access-d7dh7\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:05.041957 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.041789 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:05.041957 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.041806 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:05.041957 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.041820 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:05.041957 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.041833 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:05.041957 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.041847 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:05.041957 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.041861 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:05.041957 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.041876 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:05.050028 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.045948 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-envoy" pod "2eace1e0-9882-47ec-8c5d-927d8de6ffe6" (UID: "2eace1e0-9882-47ec-8c5d-927d8de6ffe6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:05.143436 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.143355 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eace1e0-9882-47ec-8c5d-927d8de6ffe6-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:05.160982 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.160862 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d"] Oct 30 21:00:05.164690 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.164653 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 21:00:05.166131 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.166111 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 21:00:05.589791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.588341 4848 generic.go:356] "Generic (PLEG): container finished" podID="2cbd5d2a-36e7-4b27-afb0-a94a1465278b" containerID="b32e5159799467af9e3d8392909d58baef0b6ebc86204c6fbd7e8e5c2e7d4030" exitCode=0 Oct 30 21:00:05.589791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.588521 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h" event={"ID":"2cbd5d2a-36e7-4b27-afb0-a94a1465278b","Type":"ContainerDied","Data":"b32e5159799467af9e3d8392909d58baef0b6ebc86204c6fbd7e8e5c2e7d4030"} Oct 30 21:00:05.593729 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.593686 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" event={"ID":"75e137a8-1096-4138-9e30-2cce04668543","Type":"ContainerStarted","Data":"7bc275d243e35737d56010fc880bd75497c92d88ebf0b23f627a04af15cf1dfa"} Oct 30 21:00:05.596449 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.596424 4848 generic.go:356] "Generic (PLEG): container finished" podID="16ae86a2-a837-481f-8e1d-73d846addfbf" containerID="9576b230d5aa2c41c02bfdd0a047353c50644a77ef4254c7c22d0df0b09ef78e" exitCode=60 Oct 30 21:00:05.596593 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.596522 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" event={"ID":"16ae86a2-a837-481f-8e1d-73d846addfbf","Type":"ContainerDied","Data":"9576b230d5aa2c41c02bfdd0a047353c50644a77ef4254c7c22d0df0b09ef78e"} Oct 30 21:00:05.596593 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.596567 4848 scope.go:122] "RemoveContainer" containerID="982963d41113185b1fac1523b32c0b4f5c038f5696933c40e75f7f499aa06d72" Oct 30 21:00:05.597034 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.597015 4848 scope.go:122] "RemoveContainer" containerID="9576b230d5aa2c41c02bfdd0a047353c50644a77ef4254c7c22d0df0b09ef78e" Oct 30 21:00:05.597402 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:00:05.597376 4848 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw_odh-ai-gateway-infra(16ae86a2-a837-481f-8e1d-73d846addfbf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" podUID="16ae86a2-a837-481f-8e1d-73d846addfbf" Oct 30 21:00:05.600121 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.600091 4848 generic.go:356] "Generic (PLEG): container finished" podID="2eace1e0-9882-47ec-8c5d-927d8de6ffe6" containerID="7546b89e4af05979052c575f4e4dd98bc22e433460a3c9c3b2c5a8b8d592502a" exitCode=0 Oct 30 21:00:05.606729 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.600240 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" event={"ID":"2eace1e0-9882-47ec-8c5d-927d8de6ffe6","Type":"ContainerDied","Data":"7546b89e4af05979052c575f4e4dd98bc22e433460a3c9c3b2c5a8b8d592502a"} Oct 30 21:00:05.606729 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.600465 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" event={"ID":"2eace1e0-9882-47ec-8c5d-927d8de6ffe6","Type":"ContainerDied","Data":"1fd7b2a5cfe6e171fea9b6d0e7774e3fe4cc89b100a0811cbd6bc97339ed6760"} Oct 30 21:00:05.606729 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.600327 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2" Oct 30 21:00:05.620494 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.620473 4848 scope.go:122] "RemoveContainer" containerID="7546b89e4af05979052c575f4e4dd98bc22e433460a3c9c3b2c5a8b8d592502a" Oct 30 21:00:05.621266 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.621245 4848 scope.go:122] "RemoveContainer" containerID="dfa4238d98b07b69a38a235050f23db4f9f46afb9acd646157f412750c7f94d0" Oct 30 21:00:05.621427 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.621398 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" podUID="260ef59c-c837-48ca-941b-f9304c1c5d96" containerName="maas-api" containerID="cri-o://62e665b6d61305011f39f150a50150b40e62b3c7a34709424d8ac7edbc168e31" gracePeriod=30 Oct 30 21:00:05.621538 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:00:05.621513 4848 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889900-j5hb5_odh-ai-gateway-infra(d5fcaf4f-96eb-4f50-8de3-b5755dad797f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5" podUID="d5fcaf4f-96eb-4f50-8de3-b5755dad797f" Oct 30 21:00:05.690009 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.689979 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2"] Oct 30 21:00:05.694614 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.694577 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-70cf03-openshift-default-67886c6f97-bhqq2"] Oct 30 21:00:05.697523 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.697504 4848 scope.go:122] "RemoveContainer" containerID="63ea6f5c00754eacadaeefa89d30b59385ea9c1909abfcbbb1d240648729a9ec" Oct 30 21:00:05.775741 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.775702 4848 scope.go:122] "RemoveContainer" containerID="7546b89e4af05979052c575f4e4dd98bc22e433460a3c9c3b2c5a8b8d592502a" Oct 30 21:00:05.776143 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:00:05.776116 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7546b89e4af05979052c575f4e4dd98bc22e433460a3c9c3b2c5a8b8d592502a\": container with ID starting with 7546b89e4af05979052c575f4e4dd98bc22e433460a3c9c3b2c5a8b8d592502a not found: ID does not exist" containerID="7546b89e4af05979052c575f4e4dd98bc22e433460a3c9c3b2c5a8b8d592502a" Oct 30 21:00:05.776262 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.776155 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7546b89e4af05979052c575f4e4dd98bc22e433460a3c9c3b2c5a8b8d592502a"} err="failed to get container status \"7546b89e4af05979052c575f4e4dd98bc22e433460a3c9c3b2c5a8b8d592502a\": rpc error: code = NotFound desc = could not find container \"7546b89e4af05979052c575f4e4dd98bc22e433460a3c9c3b2c5a8b8d592502a\": container with ID starting with 7546b89e4af05979052c575f4e4dd98bc22e433460a3c9c3b2c5a8b8d592502a not found: ID does not exist" Oct 30 21:00:05.776262 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.776178 4848 scope.go:122] "RemoveContainer" containerID="63ea6f5c00754eacadaeefa89d30b59385ea9c1909abfcbbb1d240648729a9ec" Oct 30 21:00:05.776529 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:00:05.776509 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ea6f5c00754eacadaeefa89d30b59385ea9c1909abfcbbb1d240648729a9ec\": container with ID starting with 63ea6f5c00754eacadaeefa89d30b59385ea9c1909abfcbbb1d240648729a9ec not found: ID does not exist" containerID="63ea6f5c00754eacadaeefa89d30b59385ea9c1909abfcbbb1d240648729a9ec" Oct 30 21:00:05.776590 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:05.776536 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ea6f5c00754eacadaeefa89d30b59385ea9c1909abfcbbb1d240648729a9ec"} err="failed to get container status \"63ea6f5c00754eacadaeefa89d30b59385ea9c1909abfcbbb1d240648729a9ec\": rpc error: code = NotFound desc = could not find container \"63ea6f5c00754eacadaeefa89d30b59385ea9c1909abfcbbb1d240648729a9ec\": container with ID starting with 63ea6f5c00754eacadaeefa89d30b59385ea9c1909abfcbbb1d240648729a9ec not found: ID does not exist" Oct 30 21:00:06.080326 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.080243 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" Oct 30 21:00:06.110533 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.091524 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260ef59c-c837-48ca-941b-f9304c1c5d96-kube-api-access-xqtv5\" (UniqueName: \"kubernetes.io/projected/260ef59c-c837-48ca-941b-f9304c1c5d96-kube-api-access-xqtv5\") pod \"260ef59c-c837-48ca-941b-f9304c1c5d96\" (UID: \"260ef59c-c837-48ca-941b-f9304c1c5d96\") " Oct 30 21:00:06.144550 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.144469 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260ef59c-c837-48ca-941b-f9304c1c5d96-kube-api-access-xqtv5" pod "260ef59c-c837-48ca-941b-f9304c1c5d96" (UID: "260ef59c-c837-48ca-941b-f9304c1c5d96"). InnerVolumeSpecName "kube-api-access-xqtv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:06.214255 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.213703 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/260ef59c-c837-48ca-941b-f9304c1c5d96-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/260ef59c-c837-48ca-941b-f9304c1c5d96-maas-api-tls\") pod \"260ef59c-c837-48ca-941b-f9304c1c5d96\" (UID: \"260ef59c-c837-48ca-941b-f9304c1c5d96\") " Oct 30 21:00:06.214255 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.213750 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/260ef59c-c837-48ca-941b-f9304c1c5d96-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/260ef59c-c837-48ca-941b-f9304c1c5d96-openshift-service-ca\") pod \"260ef59c-c837-48ca-941b-f9304c1c5d96\" (UID: \"260ef59c-c837-48ca-941b-f9304c1c5d96\") " Oct 30 21:00:06.214255 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.213805 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/260ef59c-c837-48ca-941b-f9304c1c5d96-metrics-certs\" (UniqueName: \"kubernetes.io/secret/260ef59c-c837-48ca-941b-f9304c1c5d96-metrics-certs\") pod \"260ef59c-c837-48ca-941b-f9304c1c5d96\" (UID: \"260ef59c-c837-48ca-941b-f9304c1c5d96\") " Oct 30 21:00:06.214255 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.214213 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260ef59c-c837-48ca-941b-f9304c1c5d96-openshift-service-ca" pod "260ef59c-c837-48ca-941b-f9304c1c5d96" (UID: "260ef59c-c837-48ca-941b-f9304c1c5d96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:06.214552 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.214532 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqtv5\" (UniqueName: \"kubernetes.io/projected/260ef59c-c837-48ca-941b-f9304c1c5d96-kube-api-access-xqtv5\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:06.214592 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.214560 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/260ef59c-c837-48ca-941b-f9304c1c5d96-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:06.216452 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.216429 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/260ef59c-c837-48ca-941b-f9304c1c5d96-maas-api-tls" pod "260ef59c-c837-48ca-941b-f9304c1c5d96" (UID: "260ef59c-c837-48ca-941b-f9304c1c5d96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:06.218490 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.218451 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/260ef59c-c837-48ca-941b-f9304c1c5d96-metrics-certs" pod "260ef59c-c837-48ca-941b-f9304c1c5d96" (UID: "260ef59c-c837-48ca-941b-f9304c1c5d96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:06.315997 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.315964 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/260ef59c-c837-48ca-941b-f9304c1c5d96-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:06.315997 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.315994 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/260ef59c-c837-48ca-941b-f9304c1c5d96-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:06.628618 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.628582 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" event={"ID":"75e137a8-1096-4138-9e30-2cce04668543","Type":"ContainerStarted","Data":"a922bd85a6f64e9f2da7907745f2a61a5448f3d7b196ff26f0033d68e6009461"} Oct 30 21:00:06.629097 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.628688 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" Oct 30 21:00:06.630630 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.630596 4848 scope.go:122] "RemoveContainer" containerID="9576b230d5aa2c41c02bfdd0a047353c50644a77ef4254c7c22d0df0b09ef78e" Oct 30 21:00:06.630908 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:00:06.630885 4848 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw_odh-ai-gateway-infra(16ae86a2-a837-481f-8e1d-73d846addfbf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" podUID="16ae86a2-a837-481f-8e1d-73d846addfbf" Oct 30 21:00:06.632428 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.632402 4848 generic.go:356] "Generic (PLEG): container finished" podID="260ef59c-c837-48ca-941b-f9304c1c5d96" containerID="62e665b6d61305011f39f150a50150b40e62b3c7a34709424d8ac7edbc168e31" exitCode=0 Oct 30 21:00:06.632567 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.632437 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" Oct 30 21:00:06.632567 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.632475 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" event={"ID":"260ef59c-c837-48ca-941b-f9304c1c5d96","Type":"ContainerDied","Data":"62e665b6d61305011f39f150a50150b40e62b3c7a34709424d8ac7edbc168e31"} Oct 30 21:00:06.632567 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.632524 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d" event={"ID":"260ef59c-c837-48ca-941b-f9304c1c5d96","Type":"ContainerDied","Data":"19ac29bfaf731032867bc644813a2be76d6bbbd0817101a213b0588014d409bf"} Oct 30 21:00:06.632567 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.632556 4848 scope.go:122] "RemoveContainer" containerID="62e665b6d61305011f39f150a50150b40e62b3c7a34709424d8ac7edbc168e31" Oct 30 21:00:06.644453 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.644428 4848 scope.go:122] "RemoveContainer" containerID="62e665b6d61305011f39f150a50150b40e62b3c7a34709424d8ac7edbc168e31" Oct 30 21:00:06.644736 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:00:06.644705 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e665b6d61305011f39f150a50150b40e62b3c7a34709424d8ac7edbc168e31\": container with ID starting with 62e665b6d61305011f39f150a50150b40e62b3c7a34709424d8ac7edbc168e31 not found: ID does not exist" containerID="62e665b6d61305011f39f150a50150b40e62b3c7a34709424d8ac7edbc168e31" Oct 30 21:00:06.644837 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.644745 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e665b6d61305011f39f150a50150b40e62b3c7a34709424d8ac7edbc168e31"} err="failed to get container status \"62e665b6d61305011f39f150a50150b40e62b3c7a34709424d8ac7edbc168e31\": rpc error: code = NotFound desc = could not find container \"62e665b6d61305011f39f150a50150b40e62b3c7a34709424d8ac7edbc168e31\": container with ID starting with 62e665b6d61305011f39f150a50150b40e62b3c7a34709424d8ac7edbc168e31 not found: ID does not exist" Oct 30 21:00:06.649855 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.649807 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" podStartSLOduration=2.028511926 podStartE2EDuration="2.649791119s" totalImagesPullingTime="621.279193ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:06.648522096 +0000 UTC m=+1311.828418604" watchObservedRunningTime="2026-10-30 21:00:06.649791119 +0000 UTC m=+1311.829687633" Oct 30 21:00:06.687175 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.687147 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d"] Oct 30 21:00:06.692672 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.692646 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-6c1acd-958dd874f-tc55d"] Oct 30 21:00:06.952929 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:06.952639 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="260ef59c-c837-48ca-941b-f9304c1c5d96" path="/var/lib/kubelet/pods/260ef59c-c837-48ca-941b-f9304c1c5d96/volumes" Oct 30 21:00:07.126743 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.126713 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eace1e0-9882-47ec-8c5d-927d8de6ffe6" path="/var/lib/kubelet/pods/2eace1e0-9882-47ec-8c5d-927d8de6ffe6/volumes" Oct 30 21:00:07.147528 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.147499 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h" Oct 30 21:00:07.341638 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.340099 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-maas-api-xks-ca\") pod \"2cbd5d2a-36e7-4b27-afb0-a94a1465278b\" (UID: \"2cbd5d2a-36e7-4b27-afb0-a94a1465278b\") " Oct 30 21:00:07.341638 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.340388 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-maas-api-openshift-ca\") pod \"2cbd5d2a-36e7-4b27-afb0-a94a1465278b\" (UID: \"2cbd5d2a-36e7-4b27-afb0-a94a1465278b\") " Oct 30 21:00:07.341638 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.340535 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-kube-api-access-x7nfp\" (UniqueName: \"kubernetes.io/projected/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-kube-api-access-x7nfp\") pod \"2cbd5d2a-36e7-4b27-afb0-a94a1465278b\" (UID: \"2cbd5d2a-36e7-4b27-afb0-a94a1465278b\") " Oct 30 21:00:07.341638 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.340940 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-maas-api-openshift-ca" pod "2cbd5d2a-36e7-4b27-afb0-a94a1465278b" (UID: "2cbd5d2a-36e7-4b27-afb0-a94a1465278b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:07.341638 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.341258 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-maas-api-openshift-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:07.343032 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.342998 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-maas-api-xks-ca" pod "2cbd5d2a-36e7-4b27-afb0-a94a1465278b" (UID: "2cbd5d2a-36e7-4b27-afb0-a94a1465278b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:07.343652 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.343622 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-kube-api-access-x7nfp" pod "2cbd5d2a-36e7-4b27-afb0-a94a1465278b" (UID: "2cbd5d2a-36e7-4b27-afb0-a94a1465278b"). InnerVolumeSpecName "kube-api-access-x7nfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:07.411273 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.411246 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 21:00:07.442297 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.442266 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7nfp\" (UniqueName: \"kubernetes.io/projected/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-kube-api-access-x7nfp\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:07.442297 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.442296 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cbd5d2a-36e7-4b27-afb0-a94a1465278b-maas-api-xks-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:07.542960 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.542926 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0998438-c7f6-4cb0-ba25-b406d43b8959-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0998438-c7f6-4cb0-ba25-b406d43b8959-istiod-ca-cert\") pod \"b0998438-c7f6-4cb0-ba25-b406d43b8959\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " Oct 30 21:00:07.543161 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.542973 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-podinfo\") pod \"b0998438-c7f6-4cb0-ba25-b406d43b8959\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " Oct 30 21:00:07.543161 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.543007 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-workload-certs\") pod \"b0998438-c7f6-4cb0-ba25-b406d43b8959\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " Oct 30 21:00:07.543161 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.543053 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-credential-socket\") pod \"b0998438-c7f6-4cb0-ba25-b406d43b8959\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " Oct 30 21:00:07.543329 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.543213 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-token\" (UniqueName: \"kubernetes.io/projected/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-token\") pod \"b0998438-c7f6-4cb0-ba25-b406d43b8959\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " Oct 30 21:00:07.543329 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.543315 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-data\") pod \"b0998438-c7f6-4cb0-ba25-b406d43b8959\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " Oct 30 21:00:07.543433 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.543358 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-workload-certs" pod "b0998438-c7f6-4cb0-ba25-b406d43b8959" (UID: "b0998438-c7f6-4cb0-ba25-b406d43b8959"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:07.543433 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.543391 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-credential-socket" pod "b0998438-c7f6-4cb0-ba25-b406d43b8959" (UID: "b0998438-c7f6-4cb0-ba25-b406d43b8959"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:07.543578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.543547 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0998438-c7f6-4cb0-ba25-b406d43b8959-istiod-ca-cert" pod "b0998438-c7f6-4cb0-ba25-b406d43b8959" (UID: "b0998438-c7f6-4cb0-ba25-b406d43b8959"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:07.543735 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.543694 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-data" pod "b0998438-c7f6-4cb0-ba25-b406d43b8959" (UID: "b0998438-c7f6-4cb0-ba25-b406d43b8959"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:07.543735 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.543713 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0998438-c7f6-4cb0-ba25-b406d43b8959-kube-api-access-zc287\" (UniqueName: \"kubernetes.io/projected/b0998438-c7f6-4cb0-ba25-b406d43b8959-kube-api-access-zc287\") pod \"b0998438-c7f6-4cb0-ba25-b406d43b8959\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " Oct 30 21:00:07.543901 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.543800 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-workload-socket\") pod \"b0998438-c7f6-4cb0-ba25-b406d43b8959\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " Oct 30 21:00:07.543901 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.543861 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-envoy\") pod \"b0998438-c7f6-4cb0-ba25-b406d43b8959\" (UID: \"b0998438-c7f6-4cb0-ba25-b406d43b8959\") " Oct 30 21:00:07.544094 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.544068 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-workload-socket" pod "b0998438-c7f6-4cb0-ba25-b406d43b8959" (UID: "b0998438-c7f6-4cb0-ba25-b406d43b8959"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:07.544411 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.544389 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:07.544411 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.544410 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:07.544595 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.544420 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0998438-c7f6-4cb0-ba25-b406d43b8959-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:07.544595 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.544429 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:07.544595 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.544438 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:07.545391 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.545369 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-podinfo" pod "b0998438-c7f6-4cb0-ba25-b406d43b8959" (UID: "b0998438-c7f6-4cb0-ba25-b406d43b8959"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:00:07.545753 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.545733 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0998438-c7f6-4cb0-ba25-b406d43b8959-kube-api-access-zc287" pod "b0998438-c7f6-4cb0-ba25-b406d43b8959" (UID: "b0998438-c7f6-4cb0-ba25-b406d43b8959"). InnerVolumeSpecName "kube-api-access-zc287". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:07.546113 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.546095 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-envoy" pod "b0998438-c7f6-4cb0-ba25-b406d43b8959" (UID: "b0998438-c7f6-4cb0-ba25-b406d43b8959"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:07.546182 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.546136 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-token" pod "b0998438-c7f6-4cb0-ba25-b406d43b8959" (UID: "b0998438-c7f6-4cb0-ba25-b406d43b8959"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:07.640448 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.640419 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h" Oct 30 21:00:07.640448 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.640442 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h" event={"ID":"2cbd5d2a-36e7-4b27-afb0-a94a1465278b","Type":"ContainerDied","Data":"da27ef042ea2c8ad398c7e8b7acd8e684a8ebff504e5005092da9615a3f76f78"} Oct 30 21:00:07.641007 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.640491 4848 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da27ef042ea2c8ad398c7e8b7acd8e684a8ebff504e5005092da9615a3f76f78" Oct 30 21:00:07.642169 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.642140 4848 generic.go:356] "Generic (PLEG): container finished" podID="b0998438-c7f6-4cb0-ba25-b406d43b8959" containerID="6cf00af8879dcc3d0bbd9a3d31607986495738ceaf18332628ccac0cf8af7f26" exitCode=0 Oct 30 21:00:07.642310 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.642187 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" Oct 30 21:00:07.642310 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.642227 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" event={"ID":"b0998438-c7f6-4cb0-ba25-b406d43b8959","Type":"ContainerDied","Data":"6cf00af8879dcc3d0bbd9a3d31607986495738ceaf18332628ccac0cf8af7f26"} Oct 30 21:00:07.642310 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.642270 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74" event={"ID":"b0998438-c7f6-4cb0-ba25-b406d43b8959","Type":"ContainerDied","Data":"fe31bc939c4943df7ab920b5cf7484cc268e291861a38d85366117943d4dd0ef"} Oct 30 21:00:07.642310 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.642296 4848 scope.go:122] "RemoveContainer" containerID="6cf00af8879dcc3d0bbd9a3d31607986495738ceaf18332628ccac0cf8af7f26" Oct 30 21:00:07.645473 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.645446 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:07.645577 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.645479 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc287\" (UniqueName: \"kubernetes.io/projected/b0998438-c7f6-4cb0-ba25-b406d43b8959-kube-api-access-zc287\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:07.645577 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.645496 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:07.645577 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.645511 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0998438-c7f6-4cb0-ba25-b406d43b8959-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:07.660480 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.660458 4848 scope.go:122] "RemoveContainer" containerID="65b26dc1623be9f3195ac4dbca57546a9696e367a27f16f7eb8099116c3df05d" Oct 30 21:00:07.670924 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.670889 4848 scope.go:122] "RemoveContainer" containerID="6cf00af8879dcc3d0bbd9a3d31607986495738ceaf18332628ccac0cf8af7f26" Oct 30 21:00:07.671493 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:00:07.671329 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf00af8879dcc3d0bbd9a3d31607986495738ceaf18332628ccac0cf8af7f26\": container with ID starting with 6cf00af8879dcc3d0bbd9a3d31607986495738ceaf18332628ccac0cf8af7f26 not found: ID does not exist" containerID="6cf00af8879dcc3d0bbd9a3d31607986495738ceaf18332628ccac0cf8af7f26" Oct 30 21:00:07.671493 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.671399 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf00af8879dcc3d0bbd9a3d31607986495738ceaf18332628ccac0cf8af7f26"} err="failed to get container status \"6cf00af8879dcc3d0bbd9a3d31607986495738ceaf18332628ccac0cf8af7f26\": rpc error: code = NotFound desc = could not find container \"6cf00af8879dcc3d0bbd9a3d31607986495738ceaf18332628ccac0cf8af7f26\": container with ID starting with 6cf00af8879dcc3d0bbd9a3d31607986495738ceaf18332628ccac0cf8af7f26 not found: ID does not exist" Oct 30 21:00:07.671493 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.671439 4848 scope.go:122] "RemoveContainer" containerID="65b26dc1623be9f3195ac4dbca57546a9696e367a27f16f7eb8099116c3df05d" Oct 30 21:00:07.672234 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:00:07.671841 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b26dc1623be9f3195ac4dbca57546a9696e367a27f16f7eb8099116c3df05d\": container with ID starting with 65b26dc1623be9f3195ac4dbca57546a9696e367a27f16f7eb8099116c3df05d not found: ID does not exist" containerID="65b26dc1623be9f3195ac4dbca57546a9696e367a27f16f7eb8099116c3df05d" Oct 30 21:00:07.672234 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.671877 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b26dc1623be9f3195ac4dbca57546a9696e367a27f16f7eb8099116c3df05d"} err="failed to get container status \"65b26dc1623be9f3195ac4dbca57546a9696e367a27f16f7eb8099116c3df05d\": rpc error: code = NotFound desc = could not find container \"65b26dc1623be9f3195ac4dbca57546a9696e367a27f16f7eb8099116c3df05d\": container with ID starting with 65b26dc1623be9f3195ac4dbca57546a9696e367a27f16f7eb8099116c3df05d not found: ID does not exist" Oct 30 21:00:07.675036 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.675018 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74"] Oct 30 21:00:07.679579 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:07.679557 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-99b00f-openshift-default-7767c9868f-sss74"] Oct 30 21:00:07.710727 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:00:07.710691 4848 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cbd5d2a_36e7_4b27_afb0_a94a1465278b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0998438_c7f6_4cb0_ba25_b406d43b8959.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0998438_c7f6_4cb0_ba25_b406d43b8959.slice/crio-fe31bc939c4943df7ab920b5cf7484cc268e291861a38d85366117943d4dd0ef\": RecentStats: unable to find data in memory cache]" Oct 30 21:00:08.188498 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:08.188469 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v" Oct 30 21:00:08.189247 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:08.189223 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf" Oct 30 21:00:08.937140 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:08.937108 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0998438-c7f6-4cb0-ba25-b406d43b8959" path="/var/lib/kubelet/pods/b0998438-c7f6-4cb0-ba25-b406d43b8959/volumes" Oct 30 21:00:12.563587 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:12.563552 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx"] Oct 30 21:00:12.564096 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:12.563831 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" podUID="1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" containerName="istio-proxy" containerID="cri-o://fca5ab014eb2c4e1fa7839c033686750beba009607080d89085b579f8089d104" gracePeriod=30 Oct 30 21:00:12.650000 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:12.649956 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" Oct 30 21:00:13.478867 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:13.478826 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm" Oct 30 21:00:13.479056 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:13.478905 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w" Oct 30 21:00:14.930864 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:14.930828 4848 scope.go:122] "RemoveContainer" containerID="dfa4238d98b07b69a38a235050f23db4f9f46afb9acd646157f412750c7f94d0" Oct 30 21:00:15.852406 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.852371 4848 generic.go:356] "Generic (PLEG): container finished" podID="d5fcaf4f-96eb-4f50-8de3-b5755dad797f" containerID="c3e8c4fb1e7162ff36ae0d1d6068eb570e71da6e495d81ada650512f90204480" exitCode=60 Oct 30 21:00:15.852578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.852454 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5" event={"ID":"d5fcaf4f-96eb-4f50-8de3-b5755dad797f","Type":"ContainerDied","Data":"c3e8c4fb1e7162ff36ae0d1d6068eb570e71da6e495d81ada650512f90204480"} Oct 30 21:00:15.852578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.852532 4848 scope.go:122] "RemoveContainer" containerID="dfa4238d98b07b69a38a235050f23db4f9f46afb9acd646157f412750c7f94d0" Oct 30 21:00:15.853006 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.852987 4848 scope.go:122] "RemoveContainer" containerID="c3e8c4fb1e7162ff36ae0d1d6068eb570e71da6e495d81ada650512f90204480" Oct 30 21:00:15.853328 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:00:15.853305 4848 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889900-j5hb5_odh-ai-gateway-infra(d5fcaf4f-96eb-4f50-8de3-b5755dad797f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5" podUID="d5fcaf4f-96eb-4f50-8de3-b5755dad797f" Oct 30 21:00:15.925761 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.925722 4848 scope.go:122] "RemoveContainer" containerID="9576b230d5aa2c41c02bfdd0a047353c50644a77ef4254c7c22d0df0b09ef78e" Oct 30 21:00:15.941922 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.941899 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww"] Oct 30 21:00:15.942663 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.942645 4848 cpu_manager.go:414] "containerMap: removing container" podUID="2cbd5d2a-36e7-4b27-afb0-a94a1465278b" containerName="revoke-keys" Oct 30 21:00:15.942741 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.942667 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cbd5d2a-36e7-4b27-afb0-a94a1465278b" containerName="revoke-keys" Oct 30 21:00:15.942741 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.942688 4848 cpu_manager.go:414] "containerMap: removing container" podUID="2eace1e0-9882-47ec-8c5d-927d8de6ffe6" containerName="istio-proxy" Oct 30 21:00:15.942741 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.942697 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eace1e0-9882-47ec-8c5d-927d8de6ffe6" containerName="istio-proxy" Oct 30 21:00:15.942741 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.942732 4848 cpu_manager.go:414] "containerMap: removing container" podUID="b0998438-c7f6-4cb0-ba25-b406d43b8959" containerName="istio-proxy" Oct 30 21:00:15.942741 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.942739 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0998438-c7f6-4cb0-ba25-b406d43b8959" containerName="istio-proxy" Oct 30 21:00:15.943072 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.942753 4848 cpu_manager.go:414] "containerMap: removing container" podUID="2eace1e0-9882-47ec-8c5d-927d8de6ffe6" containerName="istio-proxy" Oct 30 21:00:15.943072 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.942760 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eace1e0-9882-47ec-8c5d-927d8de6ffe6" containerName="istio-proxy" Oct 30 21:00:15.943072 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.942799 4848 cpu_manager.go:414] "containerMap: removing container" podUID="260ef59c-c837-48ca-941b-f9304c1c5d96" containerName="maas-api" Oct 30 21:00:15.943072 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.942807 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="260ef59c-c837-48ca-941b-f9304c1c5d96" containerName="maas-api" Oct 30 21:00:15.943072 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.942992 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0998438-c7f6-4cb0-ba25-b406d43b8959" containerName="istio-proxy" Oct 30 21:00:15.943072 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.943013 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cbd5d2a-36e7-4b27-afb0-a94a1465278b" containerName="revoke-keys" Oct 30 21:00:15.943072 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.943023 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eace1e0-9882-47ec-8c5d-927d8de6ffe6" containerName="istio-proxy" Oct 30 21:00:15.943072 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.943037 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eace1e0-9882-47ec-8c5d-927d8de6ffe6" containerName="istio-proxy" Oct 30 21:00:15.943072 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.943049 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="260ef59c-c837-48ca-941b-f9304c1c5d96" containerName="maas-api" Oct 30 21:00:15.948160 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.948141 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww" Oct 30 21:00:15.956876 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.956845 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww"] Oct 30 21:00:15.957882 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:15.957855 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:00:16.104844 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:16.099295 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ccfd53-5388-4265-a017-22f7cfb377cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b45884-56rww\" (UID: \"21ccfd53-5388-4265-a017-22f7cfb377cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww" Oct 30 21:00:16.104844 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:16.099401 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ccfd53-5388-4265-a017-22f7cfb377cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b45884-56rww\" (UID: \"21ccfd53-5388-4265-a017-22f7cfb377cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww" Oct 30 21:00:16.104844 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:16.099509 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4mx4\" (UniqueName: \"kubernetes.io/projected/21ccfd53-5388-4265-a017-22f7cfb377cb-kube-api-access-z4mx4\") pod \"maas-api-revoke-sub-47b45884-56rww\" (UID: \"21ccfd53-5388-4265-a017-22f7cfb377cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww" Oct 30 21:00:16.200263 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:16.200223 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ccfd53-5388-4265-a017-22f7cfb377cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b45884-56rww\" (UID: \"21ccfd53-5388-4265-a017-22f7cfb377cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww" Oct 30 21:00:16.200451 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:16.200323 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ccfd53-5388-4265-a017-22f7cfb377cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b45884-56rww\" (UID: \"21ccfd53-5388-4265-a017-22f7cfb377cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww" Oct 30 21:00:16.200451 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:16.200403 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4mx4\" (UniqueName: \"kubernetes.io/projected/21ccfd53-5388-4265-a017-22f7cfb377cb-kube-api-access-z4mx4\") pod \"maas-api-revoke-sub-47b45884-56rww\" (UID: \"21ccfd53-5388-4265-a017-22f7cfb377cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww" Oct 30 21:00:16.201064 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:16.201035 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ccfd53-5388-4265-a017-22f7cfb377cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47b45884-56rww\" (UID: \"21ccfd53-5388-4265-a017-22f7cfb377cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww" Oct 30 21:00:16.202783 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:16.202747 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ccfd53-5388-4265-a017-22f7cfb377cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47b45884-56rww\" (UID: \"21ccfd53-5388-4265-a017-22f7cfb377cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww" Oct 30 21:00:16.208694 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:16.208668 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4mx4\" (UniqueName: \"kubernetes.io/projected/21ccfd53-5388-4265-a017-22f7cfb377cb-kube-api-access-z4mx4\") pod \"maas-api-revoke-sub-47b45884-56rww\" (UID: \"21ccfd53-5388-4265-a017-22f7cfb377cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww" Oct 30 21:00:16.281810 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:16.281753 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww" Oct 30 21:00:16.578837 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:16.578367 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww"] Oct 30 21:00:16.578837 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:00:16.578454 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ccfd53_5388_4265_a017_22f7cfb377cb.slice/crio-c8e07a91b2c7ea8260bedeb5864eba0382dbf44dce5089e90aa538da6ced342d WatchSource:0}: Error finding container c8e07a91b2c7ea8260bedeb5864eba0382dbf44dce5089e90aa538da6ced342d: Status 404 returned error can't find the container with id c8e07a91b2c7ea8260bedeb5864eba0382dbf44dce5089e90aa538da6ced342d Oct 30 21:00:16.898411 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:16.898373 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww" event={"ID":"21ccfd53-5388-4265-a017-22f7cfb377cb","Type":"ContainerStarted","Data":"c8e07a91b2c7ea8260bedeb5864eba0382dbf44dce5089e90aa538da6ced342d"} Oct 30 21:00:16.904834 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:16.903218 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5"] Oct 30 21:00:16.904834 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:16.903813 4848 generic.go:356] "Generic (PLEG): container finished" podID="16ae86a2-a837-481f-8e1d-73d846addfbf" containerID="ecf7c88e6a17edfb69c438d76d6d2afb081f31c8d6b0c1e7b72d0a115780d11c" exitCode=60 Oct 30 21:00:16.904834 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:16.903908 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" event={"ID":"16ae86a2-a837-481f-8e1d-73d846addfbf","Type":"ContainerDied","Data":"ecf7c88e6a17edfb69c438d76d6d2afb081f31c8d6b0c1e7b72d0a115780d11c"} Oct 30 21:00:16.904834 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:16.903960 4848 scope.go:122] "RemoveContainer" containerID="9576b230d5aa2c41c02bfdd0a047353c50644a77ef4254c7c22d0df0b09ef78e" Oct 30 21:00:16.904834 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:16.904436 4848 scope.go:122] "RemoveContainer" containerID="ecf7c88e6a17edfb69c438d76d6d2afb081f31c8d6b0c1e7b72d0a115780d11c" Oct 30 21:00:16.907694 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:00:16.904982 4848 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw_odh-ai-gateway-infra(16ae86a2-a837-481f-8e1d-73d846addfbf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" podUID="16ae86a2-a837-481f-8e1d-73d846addfbf" Oct 30 21:00:17.050195 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:17.050172 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5" Oct 30 21:00:17.215458 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:17.215431 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5fcaf4f-96eb-4f50-8de3-b5755dad797f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5fcaf4f-96eb-4f50-8de3-b5755dad797f-maas-api-service-ca\") pod \"d5fcaf4f-96eb-4f50-8de3-b5755dad797f\" (UID: \"d5fcaf4f-96eb-4f50-8de3-b5755dad797f\") " Oct 30 21:00:17.215609 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:17.215525 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5fcaf4f-96eb-4f50-8de3-b5755dad797f-kube-api-access-pqg6q\" (UniqueName: \"kubernetes.io/projected/d5fcaf4f-96eb-4f50-8de3-b5755dad797f-kube-api-access-pqg6q\") pod \"d5fcaf4f-96eb-4f50-8de3-b5755dad797f\" (UID: \"d5fcaf4f-96eb-4f50-8de3-b5755dad797f\") " Oct 30 21:00:17.215840 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:17.215797 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5fcaf4f-96eb-4f50-8de3-b5755dad797f-maas-api-service-ca" pod "d5fcaf4f-96eb-4f50-8de3-b5755dad797f" (UID: "d5fcaf4f-96eb-4f50-8de3-b5755dad797f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:17.216184 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:17.216166 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5fcaf4f-96eb-4f50-8de3-b5755dad797f-maas-api-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:17.217704 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:17.217682 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5fcaf4f-96eb-4f50-8de3-b5755dad797f-kube-api-access-pqg6q" pod "d5fcaf4f-96eb-4f50-8de3-b5755dad797f" (UID: "d5fcaf4f-96eb-4f50-8de3-b5755dad797f"). InnerVolumeSpecName "kube-api-access-pqg6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:17.317029 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:17.317000 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqg6q\" (UniqueName: \"kubernetes.io/projected/d5fcaf4f-96eb-4f50-8de3-b5755dad797f-kube-api-access-pqg6q\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:17.920847 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:17.920730 4848 generic.go:356] "Generic (PLEG): container finished" podID="21ccfd53-5388-4265-a017-22f7cfb377cb" containerID="82830b81f5a97afa4f485c72cc06925007bc05a5ad14c53c5c064ac5108984d7" exitCode=0 Oct 30 21:00:17.921009 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:17.920856 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww" event={"ID":"21ccfd53-5388-4265-a017-22f7cfb377cb","Type":"ContainerDied","Data":"82830b81f5a97afa4f485c72cc06925007bc05a5ad14c53c5c064ac5108984d7"} Oct 30 21:00:17.925304 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:17.924549 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5" event={"ID":"d5fcaf4f-96eb-4f50-8de3-b5755dad797f","Type":"ContainerDied","Data":"35c036186a12bc0b7cc3c53307d1d74cabeda8f30b3b113b2509baee2b2c62ca"} Oct 30 21:00:17.925304 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:17.924614 4848 scope.go:122] "RemoveContainer" containerID="c3e8c4fb1e7162ff36ae0d1d6068eb570e71da6e495d81ada650512f90204480" Oct 30 21:00:17.925304 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:17.924741 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5" Oct 30 21:00:17.956487 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:17.952065 4848 generic.go:356] "Generic (PLEG): container finished" podID="1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" containerID="fca5ab014eb2c4e1fa7839c033686750beba009607080d89085b579f8089d104" exitCode=0 Oct 30 21:00:17.956487 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:17.952145 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" event={"ID":"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e","Type":"ContainerDied","Data":"fca5ab014eb2c4e1fa7839c033686750beba009607080d89085b579f8089d104"} Oct 30 21:00:17.956487 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:17.956465 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw"] Oct 30 21:00:17.969911 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:17.969880 4848 scope.go:122] "RemoveContainer" containerID="2d358be9ef0986f7d0e7776ea57191f5e2a4aa52162894a3138941fb7128cceb" Oct 30 21:00:17.980959 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:17.980934 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 21:00:18.015831 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.013211 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5"] Oct 30 21:00:18.020816 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.019517 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889900-j5hb5"] Oct 30 21:00:18.052200 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.051808 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-token\" (UniqueName: \"kubernetes.io/projected/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-token\") pod \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " Oct 30 21:00:18.052200 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.051924 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-envoy\") pod \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " Oct 30 21:00:18.052200 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.051964 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-data\") pod \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " Oct 30 21:00:18.052200 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.051993 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-credential-socket\") pod \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " Oct 30 21:00:18.052200 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.052086 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-kube-api-access-fhk6k\" (UniqueName: \"kubernetes.io/projected/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-kube-api-access-fhk6k\") pod \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " Oct 30 21:00:18.052200 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.052122 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-workload-socket\") pod \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " Oct 30 21:00:18.053994 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.053202 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-podinfo\") pod \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " Oct 30 21:00:18.053994 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.053258 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istiod-ca-cert\") pod \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " Oct 30 21:00:18.053994 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.053293 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-workload-certs\") pod \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\" (UID: \"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e\") " Oct 30 21:00:18.054297 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.054241 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-workload-certs" pod "1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" (UID: "1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:18.056705 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.056166 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-workload-socket" pod "1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" (UID: "1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:18.063468 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.061286 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-credential-socket" pod "1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" (UID: "1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:18.063468 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.061312 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istiod-ca-cert" pod "1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" (UID: "1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:18.063468 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.061400 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-data" pod "1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" (UID: "1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:18.063468 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.061850 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-kube-api-access-fhk6k" pod "1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" (UID: "1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e"). InnerVolumeSpecName "kube-api-access-fhk6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:18.063468 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.062664 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-podinfo" pod "1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" (UID: "1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:00:18.063468 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.062704 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-envoy" pod "1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" (UID: "1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:18.063468 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.063429 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-token" pod "1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" (UID: "1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:18.155059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.155015 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhk6k\" (UniqueName: \"kubernetes.io/projected/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-kube-api-access-fhk6k\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:18.155059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.155047 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:18.155059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.155056 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:18.155059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.155065 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:18.155400 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.155074 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:18.155400 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.155083 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:18.155400 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.155091 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:18.155400 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.155099 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:18.155400 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.155106 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:18.176843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.176822 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" Oct 30 21:00:18.256684 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.256639 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16ae86a2-a837-481f-8e1d-73d846addfbf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ae86a2-a837-481f-8e1d-73d846addfbf-maas-api-service-ca\") pod \"16ae86a2-a837-481f-8e1d-73d846addfbf\" (UID: \"16ae86a2-a837-481f-8e1d-73d846addfbf\") " Oct 30 21:00:18.257394 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.257351 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ae86a2-a837-481f-8e1d-73d846addfbf-maas-api-service-ca" pod "16ae86a2-a837-481f-8e1d-73d846addfbf" (UID: "16ae86a2-a837-481f-8e1d-73d846addfbf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:18.358062 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.358026 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16ae86a2-a837-481f-8e1d-73d846addfbf-kube-api-access-fxdwm\" (UniqueName: \"kubernetes.io/projected/16ae86a2-a837-481f-8e1d-73d846addfbf-kube-api-access-fxdwm\") pod \"16ae86a2-a837-481f-8e1d-73d846addfbf\" (UID: \"16ae86a2-a837-481f-8e1d-73d846addfbf\") " Oct 30 21:00:18.358560 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.358540 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16ae86a2-a837-481f-8e1d-73d846addfbf-maas-api-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:18.360140 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.360114 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ae86a2-a837-481f-8e1d-73d846addfbf-kube-api-access-fxdwm" pod "16ae86a2-a837-481f-8e1d-73d846addfbf" (UID: "16ae86a2-a837-481f-8e1d-73d846addfbf"). InnerVolumeSpecName "kube-api-access-fxdwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:18.459977 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.459895 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxdwm\" (UniqueName: \"kubernetes.io/projected/16ae86a2-a837-481f-8e1d-73d846addfbf-kube-api-access-fxdwm\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:18.582383 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.582350 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v"] Oct 30 21:00:18.582950 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.582935 4848 cpu_manager.go:414] "containerMap: removing container" podUID="1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" containerName="istio-proxy" Oct 30 21:00:18.583008 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.582950 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" containerName="istio-proxy" Oct 30 21:00:18.583008 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.582967 4848 cpu_manager.go:414] "containerMap: removing container" podUID="16ae86a2-a837-481f-8e1d-73d846addfbf" containerName="cleanup" Oct 30 21:00:18.583008 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.582972 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ae86a2-a837-481f-8e1d-73d846addfbf" containerName="cleanup" Oct 30 21:00:18.583008 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.582981 4848 cpu_manager.go:414] "containerMap: removing container" podUID="b0998438-c7f6-4cb0-ba25-b406d43b8959" containerName="istio-proxy" Oct 30 21:00:18.583008 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.582986 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0998438-c7f6-4cb0-ba25-b406d43b8959" containerName="istio-proxy" Oct 30 21:00:18.583008 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.583008 4848 cpu_manager.go:414] "containerMap: removing container" podUID="16ae86a2-a837-481f-8e1d-73d846addfbf" containerName="cleanup" Oct 30 21:00:18.583312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.583013 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ae86a2-a837-481f-8e1d-73d846addfbf" containerName="cleanup" Oct 30 21:00:18.583312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.583024 4848 cpu_manager.go:414] "containerMap: removing container" podUID="d5fcaf4f-96eb-4f50-8de3-b5755dad797f" containerName="cleanup" Oct 30 21:00:18.583312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.583028 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5fcaf4f-96eb-4f50-8de3-b5755dad797f" containerName="cleanup" Oct 30 21:00:18.583312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.583036 4848 cpu_manager.go:414] "containerMap: removing container" podUID="d5fcaf4f-96eb-4f50-8de3-b5755dad797f" containerName="cleanup" Oct 30 21:00:18.583312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.583040 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5fcaf4f-96eb-4f50-8de3-b5755dad797f" containerName="cleanup" Oct 30 21:00:18.583312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.583068 4848 cpu_manager.go:414] "containerMap: removing container" podUID="16ae86a2-a837-481f-8e1d-73d846addfbf" containerName="cleanup" Oct 30 21:00:18.583312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.583074 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ae86a2-a837-481f-8e1d-73d846addfbf" containerName="cleanup" Oct 30 21:00:18.583312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.583082 4848 cpu_manager.go:414] "containerMap: removing container" podUID="d5fcaf4f-96eb-4f50-8de3-b5755dad797f" containerName="cleanup" Oct 30 21:00:18.583312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.583089 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5fcaf4f-96eb-4f50-8de3-b5755dad797f" containerName="cleanup" Oct 30 21:00:18.583312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.583178 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5fcaf4f-96eb-4f50-8de3-b5755dad797f" containerName="cleanup" Oct 30 21:00:18.583312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.583193 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" containerName="istio-proxy" Oct 30 21:00:18.583312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.583200 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ae86a2-a837-481f-8e1d-73d846addfbf" containerName="cleanup" Oct 30 21:00:18.583312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.583210 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0998438-c7f6-4cb0-ba25-b406d43b8959" containerName="istio-proxy" Oct 30 21:00:18.583312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.583218 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" containerName="istio-proxy" Oct 30 21:00:18.583312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.583224 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5fcaf4f-96eb-4f50-8de3-b5755dad797f" containerName="cleanup" Oct 30 21:00:18.583312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.583232 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ae86a2-a837-481f-8e1d-73d846addfbf" containerName="cleanup" Oct 30 21:00:18.586805 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.586785 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v" Oct 30 21:00:18.595350 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.595124 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v"] Oct 30 21:00:18.662724 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.662680 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjjfz\" (UniqueName: \"kubernetes.io/projected/c72c5dba-3c91-48e6-90ea-b28ec0a23538-kube-api-access-gjjfz\") pod \"maas-api-revoke-sub-420b975b-q6c2v\" (UID: \"c72c5dba-3c91-48e6-90ea-b28ec0a23538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v" Oct 30 21:00:18.662926 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.662856 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c72c5dba-3c91-48e6-90ea-b28ec0a23538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-420b975b-q6c2v\" (UID: \"c72c5dba-3c91-48e6-90ea-b28ec0a23538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v" Oct 30 21:00:18.663045 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.663017 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c72c5dba-3c91-48e6-90ea-b28ec0a23538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-420b975b-q6c2v\" (UID: \"c72c5dba-3c91-48e6-90ea-b28ec0a23538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v" Oct 30 21:00:18.766703 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.766611 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c72c5dba-3c91-48e6-90ea-b28ec0a23538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-420b975b-q6c2v\" (UID: \"c72c5dba-3c91-48e6-90ea-b28ec0a23538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v" Oct 30 21:00:18.767062 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.767039 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjjfz\" (UniqueName: \"kubernetes.io/projected/c72c5dba-3c91-48e6-90ea-b28ec0a23538-kube-api-access-gjjfz\") pod \"maas-api-revoke-sub-420b975b-q6c2v\" (UID: \"c72c5dba-3c91-48e6-90ea-b28ec0a23538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v" Oct 30 21:00:18.767609 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.767579 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c72c5dba-3c91-48e6-90ea-b28ec0a23538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-420b975b-q6c2v\" (UID: \"c72c5dba-3c91-48e6-90ea-b28ec0a23538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v" Oct 30 21:00:18.771816 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.767594 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c72c5dba-3c91-48e6-90ea-b28ec0a23538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-420b975b-q6c2v\" (UID: \"c72c5dba-3c91-48e6-90ea-b28ec0a23538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v" Oct 30 21:00:18.796504 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.796436 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c72c5dba-3c91-48e6-90ea-b28ec0a23538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-420b975b-q6c2v\" (UID: \"c72c5dba-3c91-48e6-90ea-b28ec0a23538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v" Oct 30 21:00:18.806204 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.802481 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjjfz\" (UniqueName: \"kubernetes.io/projected/c72c5dba-3c91-48e6-90ea-b28ec0a23538-kube-api-access-gjjfz\") pod \"maas-api-revoke-sub-420b975b-q6c2v\" (UID: \"c72c5dba-3c91-48e6-90ea-b28ec0a23538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v" Oct 30 21:00:18.869930 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.869889 4848 patch_prober.go:29] interesting pod/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:00:18.870179 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.870153 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" podUID="1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:18.902716 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.902660 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v" Oct 30 21:00:18.941360 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.941322 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5fcaf4f-96eb-4f50-8de3-b5755dad797f" path="/var/lib/kubelet/pods/d5fcaf4f-96eb-4f50-8de3-b5755dad797f/volumes" Oct 30 21:00:18.968002 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.967966 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" event={"ID":"16ae86a2-a837-481f-8e1d-73d846addfbf","Type":"ContainerDied","Data":"6e27de1fb63e0070649087dc82d5c80327d863aa06f4d4a57468df5ed550ed77"} Oct 30 21:00:18.968135 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.968039 4848 scope.go:122] "RemoveContainer" containerID="ecf7c88e6a17edfb69c438d76d6d2afb081f31c8d6b0c1e7b72d0a115780d11c" Oct 30 21:00:18.968200 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.968150 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw" Oct 30 21:00:18.974576 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.974549 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" event={"ID":"1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e","Type":"ContainerDied","Data":"34f5a948c7850572920ec7690432ba3d24ca07ce4a6f179e4f6ed33885b2c975"} Oct 30 21:00:18.974749 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:18.974691 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx" Oct 30 21:00:19.006872 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:19.006841 4848 scope.go:122] "RemoveContainer" containerID="fca5ab014eb2c4e1fa7839c033686750beba009607080d89085b579f8089d104" Oct 30 21:00:19.116129 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:19.113864 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw"] Oct 30 21:00:19.121110 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:19.121055 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-53aa47-29889900-n5hbw"] Oct 30 21:00:19.131469 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:19.131400 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx"] Oct 30 21:00:19.135682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:19.135635 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-6c1acd-openshift-default-7d464959bf-2nfzx"] Oct 30 21:00:19.287964 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:19.287655 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v"] Oct 30 21:00:19.501171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:19.501138 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf"] Oct 30 21:00:19.600278 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:19.594123 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v"] Oct 30 21:00:20.349118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:20.349086 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww" Oct 30 21:00:20.439205 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:20.439169 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ccfd53-5388-4265-a017-22f7cfb377cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ccfd53-5388-4265-a017-22f7cfb377cb-maas-api-xks-ca\") pod \"21ccfd53-5388-4265-a017-22f7cfb377cb\" (UID: \"21ccfd53-5388-4265-a017-22f7cfb377cb\") " Oct 30 21:00:20.504334 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:20.504232 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ccfd53-5388-4265-a017-22f7cfb377cb-kube-api-access-z4mx4\" (UniqueName: \"kubernetes.io/projected/21ccfd53-5388-4265-a017-22f7cfb377cb-kube-api-access-z4mx4\") pod \"21ccfd53-5388-4265-a017-22f7cfb377cb\" (UID: \"21ccfd53-5388-4265-a017-22f7cfb377cb\") " Oct 30 21:00:20.504707 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:20.504682 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ccfd53-5388-4265-a017-22f7cfb377cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ccfd53-5388-4265-a017-22f7cfb377cb-maas-api-openshift-ca\") pod \"21ccfd53-5388-4265-a017-22f7cfb377cb\" (UID: \"21ccfd53-5388-4265-a017-22f7cfb377cb\") " Oct 30 21:00:20.505737 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:20.505712 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ccfd53-5388-4265-a017-22f7cfb377cb-maas-api-openshift-ca" pod "21ccfd53-5388-4265-a017-22f7cfb377cb" (UID: "21ccfd53-5388-4265-a017-22f7cfb377cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:20.562148 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:20.530941 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ccfd53-5388-4265-a017-22f7cfb377cb-maas-api-xks-ca" pod "21ccfd53-5388-4265-a017-22f7cfb377cb" (UID: "21ccfd53-5388-4265-a017-22f7cfb377cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:20.562148 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:20.541920 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ccfd53-5388-4265-a017-22f7cfb377cb-kube-api-access-z4mx4" pod "21ccfd53-5388-4265-a017-22f7cfb377cb" (UID: "21ccfd53-5388-4265-a017-22f7cfb377cb"). InnerVolumeSpecName "kube-api-access-z4mx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:20.562148 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:20.562014 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww" Oct 30 21:00:20.567610 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:20.567566 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww" event={"ID":"21ccfd53-5388-4265-a017-22f7cfb377cb","Type":"ContainerDied","Data":"c8e07a91b2c7ea8260bedeb5864eba0382dbf44dce5089e90aa538da6ced342d"} Oct 30 21:00:20.567827 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:20.567811 4848 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8e07a91b2c7ea8260bedeb5864eba0382dbf44dce5089e90aa538da6ced342d" Oct 30 21:00:20.567956 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:20.567941 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v" event={"ID":"c72c5dba-3c91-48e6-90ea-b28ec0a23538","Type":"ContainerStarted","Data":"547c54fe964a239b460c17747e58a8f3b8559bb987653bb8e9d4f6e8a0559736"} Oct 30 21:00:20.600939 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:20.600681 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fztgr_bce11e0b-618d-40b0-9709-d6b3deda2012/payload-processing/0.log" Oct 30 21:00:20.651792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:20.634190 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4mx4\" (UniqueName: \"kubernetes.io/projected/21ccfd53-5388-4265-a017-22f7cfb377cb-kube-api-access-z4mx4\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:20.652160 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:20.652136 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ccfd53-5388-4265-a017-22f7cfb377cb-maas-api-openshift-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:20.652408 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:20.652390 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ccfd53-5388-4265-a017-22f7cfb377cb-maas-api-xks-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:20.831856 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:20.829111 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fztgr_bce11e0b-618d-40b0-9709-d6b3deda2012/payload-processing/0.log" Oct 30 21:00:21.022790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:21.021846 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16ae86a2-a837-481f-8e1d-73d846addfbf" path="/var/lib/kubelet/pods/16ae86a2-a837-481f-8e1d-73d846addfbf/volumes" Oct 30 21:00:21.066470 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:21.066439 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm_e1246b25-a2e4-4021-b38e-64a255ba4a39/payload-processing/0.log" Oct 30 21:00:21.081027 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:21.080995 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" path="/var/lib/kubelet/pods/1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e/volumes" Oct 30 21:00:21.577559 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:21.577515 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v" event={"ID":"c72c5dba-3c91-48e6-90ea-b28ec0a23538","Type":"ContainerStarted","Data":"40d2459d72c7ff4a8f2bc73651b8ab47a47d5178729400cf653a58ec73335ab2"} Oct 30 21:00:21.594624 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:21.594572 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v" podStartSLOduration=3.5945593110000003 podStartE2EDuration="3.594559311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:21.592800545 +0000 UTC m=+1326.772697054" watchObservedRunningTime="2026-10-30 21:00:21.594559311 +0000 UTC m=+1326.774455853" Oct 30 21:00:22.536102 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:00:22.536060 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:00:22.585230 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:22.585199 4848 generic.go:356] "Generic (PLEG): container finished" podID="c72c5dba-3c91-48e6-90ea-b28ec0a23538" containerID="40d2459d72c7ff4a8f2bc73651b8ab47a47d5178729400cf653a58ec73335ab2" exitCode=0 Oct 30 21:00:22.585623 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:22.585261 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v" event={"ID":"c72c5dba-3c91-48e6-90ea-b28ec0a23538","Type":"ContainerDied","Data":"40d2459d72c7ff4a8f2bc73651b8ab47a47d5178729400cf653a58ec73335ab2"} Oct 30 21:00:23.790432 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:23.790406 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v" Oct 30 21:00:23.888415 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:23.888329 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c72c5dba-3c91-48e6-90ea-b28ec0a23538-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c72c5dba-3c91-48e6-90ea-b28ec0a23538-maas-api-xks-ca\") pod \"c72c5dba-3c91-48e6-90ea-b28ec0a23538\" (UID: \"c72c5dba-3c91-48e6-90ea-b28ec0a23538\") " Oct 30 21:00:23.888593 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:23.888463 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c72c5dba-3c91-48e6-90ea-b28ec0a23538-kube-api-access-gjjfz\" (UniqueName: \"kubernetes.io/projected/c72c5dba-3c91-48e6-90ea-b28ec0a23538-kube-api-access-gjjfz\") pod \"c72c5dba-3c91-48e6-90ea-b28ec0a23538\" (UID: \"c72c5dba-3c91-48e6-90ea-b28ec0a23538\") " Oct 30 21:00:23.888593 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:23.888525 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c72c5dba-3c91-48e6-90ea-b28ec0a23538-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c72c5dba-3c91-48e6-90ea-b28ec0a23538-maas-api-openshift-ca\") pod \"c72c5dba-3c91-48e6-90ea-b28ec0a23538\" (UID: \"c72c5dba-3c91-48e6-90ea-b28ec0a23538\") " Oct 30 21:00:23.888936 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:23.888911 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72c5dba-3c91-48e6-90ea-b28ec0a23538-maas-api-openshift-ca" pod "c72c5dba-3c91-48e6-90ea-b28ec0a23538" (UID: "c72c5dba-3c91-48e6-90ea-b28ec0a23538"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:23.889146 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:23.889129 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c72c5dba-3c91-48e6-90ea-b28ec0a23538-maas-api-openshift-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:23.890548 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:23.890521 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c72c5dba-3c91-48e6-90ea-b28ec0a23538-maas-api-xks-ca" pod "c72c5dba-3c91-48e6-90ea-b28ec0a23538" (UID: "c72c5dba-3c91-48e6-90ea-b28ec0a23538"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:23.890650 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:23.890583 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72c5dba-3c91-48e6-90ea-b28ec0a23538-kube-api-access-gjjfz" pod "c72c5dba-3c91-48e6-90ea-b28ec0a23538" (UID: "c72c5dba-3c91-48e6-90ea-b28ec0a23538"). InnerVolumeSpecName "kube-api-access-gjjfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:23.990379 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:23.990344 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjjfz\" (UniqueName: \"kubernetes.io/projected/c72c5dba-3c91-48e6-90ea-b28ec0a23538-kube-api-access-gjjfz\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:23.990379 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:23.990379 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c72c5dba-3c91-48e6-90ea-b28ec0a23538-maas-api-xks-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:24.516897 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:24.516848 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf" podUID="8d53735c-521f-45e0-9653-84d72efd49e2" containerName="payload-processing" containerID="cri-o://49d64ea267963f68ca859917eea640fd01b3eb5cf30134305c042fbbce828ed9" gracePeriod=25 Oct 30 21:00:24.594443 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:24.594387 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v" podUID="54e85613-de9d-471e-b363-0eeb8c7972fe" containerName="payload-pre-processing" containerID="cri-o://3f50f3509d183e2604e8840370ca8ca3ce11ef494ca7fec4c9428ec159ef8081" gracePeriod=25 Oct 30 21:00:24.600970 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:24.600943 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v" Oct 30 21:00:24.601155 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:24.600993 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v" event={"ID":"c72c5dba-3c91-48e6-90ea-b28ec0a23538","Type":"ContainerDied","Data":"547c54fe964a239b460c17747e58a8f3b8559bb987653bb8e9d4f6e8a0559736"} Oct 30 21:00:24.601155 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:24.601061 4848 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="547c54fe964a239b460c17747e58a8f3b8559bb987653bb8e9d4f6e8a0559736" Oct 30 21:00:24.944440 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:24.944203 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v" Oct 30 21:00:25.024416 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.024344 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54e85613-de9d-471e-b363-0eeb8c7972fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54e85613-de9d-471e-b363-0eeb8c7972fe-plugins-config-volume\") pod \"54e85613-de9d-471e-b363-0eeb8c7972fe\" (UID: \"54e85613-de9d-471e-b363-0eeb8c7972fe\") " Oct 30 21:00:25.024926 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.024800 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e85613-de9d-471e-b363-0eeb8c7972fe-plugins-config-volume" pod "54e85613-de9d-471e-b363-0eeb8c7972fe" (UID: "54e85613-de9d-471e-b363-0eeb8c7972fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:25.025192 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.025170 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e85613-de9d-471e-b363-0eeb8c7972fe-kube-api-access-nt9vb\" (UniqueName: \"kubernetes.io/projected/54e85613-de9d-471e-b363-0eeb8c7972fe-kube-api-access-nt9vb\") pod \"54e85613-de9d-471e-b363-0eeb8c7972fe\" (UID: \"54e85613-de9d-471e-b363-0eeb8c7972fe\") " Oct 30 21:00:25.026595 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.026079 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54e85613-de9d-471e-b363-0eeb8c7972fe-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:25.046716 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.046672 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e85613-de9d-471e-b363-0eeb8c7972fe-kube-api-access-nt9vb" pod "54e85613-de9d-471e-b363-0eeb8c7972fe" (UID: "54e85613-de9d-471e-b363-0eeb8c7972fe"). InnerVolumeSpecName "kube-api-access-nt9vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:25.061166 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.061136 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf" Oct 30 21:00:25.127317 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.127284 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d53735c-521f-45e0-9653-84d72efd49e2-kube-api-access-smp9f\" (UniqueName: \"kubernetes.io/projected/8d53735c-521f-45e0-9653-84d72efd49e2-kube-api-access-smp9f\") pod \"8d53735c-521f-45e0-9653-84d72efd49e2\" (UID: \"8d53735c-521f-45e0-9653-84d72efd49e2\") " Oct 30 21:00:25.127459 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.127372 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d53735c-521f-45e0-9653-84d72efd49e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d53735c-521f-45e0-9653-84d72efd49e2-plugins-config-volume\") pod \"8d53735c-521f-45e0-9653-84d72efd49e2\" (UID: \"8d53735c-521f-45e0-9653-84d72efd49e2\") " Oct 30 21:00:25.127784 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.127747 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d53735c-521f-45e0-9653-84d72efd49e2-plugins-config-volume" pod "8d53735c-521f-45e0-9653-84d72efd49e2" (UID: "8d53735c-521f-45e0-9653-84d72efd49e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:25.127906 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.127841 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt9vb\" (UniqueName: \"kubernetes.io/projected/54e85613-de9d-471e-b363-0eeb8c7972fe-kube-api-access-nt9vb\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:25.129239 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.129215 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d53735c-521f-45e0-9653-84d72efd49e2-kube-api-access-smp9f" pod "8d53735c-521f-45e0-9653-84d72efd49e2" (UID: "8d53735c-521f-45e0-9653-84d72efd49e2"). InnerVolumeSpecName "kube-api-access-smp9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:25.229070 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.229039 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smp9f\" (UniqueName: \"kubernetes.io/projected/8d53735c-521f-45e0-9653-84d72efd49e2-kube-api-access-smp9f\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:25.229070 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.229066 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d53735c-521f-45e0-9653-84d72efd49e2-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:25.610071 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.610038 4848 generic.go:356] "Generic (PLEG): container finished" podID="54e85613-de9d-471e-b363-0eeb8c7972fe" containerID="3f50f3509d183e2604e8840370ca8ca3ce11ef494ca7fec4c9428ec159ef8081" exitCode=0 Oct 30 21:00:25.610280 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.610087 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v" Oct 30 21:00:25.610280 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.610122 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v" event={"ID":"54e85613-de9d-471e-b363-0eeb8c7972fe","Type":"ContainerDied","Data":"3f50f3509d183e2604e8840370ca8ca3ce11ef494ca7fec4c9428ec159ef8081"} Oct 30 21:00:25.610280 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.610188 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v" event={"ID":"54e85613-de9d-471e-b363-0eeb8c7972fe","Type":"ContainerDied","Data":"8011c0549e2a3b31bf19127c756189d89b6164559889d649f650d495ade91e75"} Oct 30 21:00:25.610280 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.610219 4848 scope.go:122] "RemoveContainer" containerID="3f50f3509d183e2604e8840370ca8ca3ce11ef494ca7fec4c9428ec159ef8081" Oct 30 21:00:25.613186 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.613007 4848 generic.go:356] "Generic (PLEG): container finished" podID="8d53735c-521f-45e0-9653-84d72efd49e2" containerID="49d64ea267963f68ca859917eea640fd01b3eb5cf30134305c042fbbce828ed9" exitCode=0 Oct 30 21:00:25.613186 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.613052 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf" event={"ID":"8d53735c-521f-45e0-9653-84d72efd49e2","Type":"ContainerDied","Data":"49d64ea267963f68ca859917eea640fd01b3eb5cf30134305c042fbbce828ed9"} Oct 30 21:00:25.613186 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.613072 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf" Oct 30 21:00:25.613186 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.613113 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf" event={"ID":"8d53735c-521f-45e0-9653-84d72efd49e2","Type":"ContainerDied","Data":"bf2356b38213d915edbce556f716ab34506b622c42a79c942c6de3dca5b6b5f3"} Oct 30 21:00:25.616617 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.616593 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq"] Oct 30 21:00:25.617556 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.617533 4848 cpu_manager.go:414] "containerMap: removing container" podUID="1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" containerName="istio-proxy" Oct 30 21:00:25.617721 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.617698 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e08fcfc-f83f-4d7a-9ee2-cc7657244a4e" containerName="istio-proxy" Oct 30 21:00:25.617855 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.617728 4848 cpu_manager.go:414] "containerMap: removing container" podUID="54e85613-de9d-471e-b363-0eeb8c7972fe" containerName="payload-pre-processing" Oct 30 21:00:25.617855 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.617740 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e85613-de9d-471e-b363-0eeb8c7972fe" containerName="payload-pre-processing" Oct 30 21:00:25.617855 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.617755 4848 cpu_manager.go:414] "containerMap: removing container" podUID="c72c5dba-3c91-48e6-90ea-b28ec0a23538" containerName="revoke-keys" Oct 30 21:00:25.617855 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.617778 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72c5dba-3c91-48e6-90ea-b28ec0a23538" containerName="revoke-keys" Oct 30 21:00:25.617855 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.617794 4848 cpu_manager.go:414] "containerMap: removing container" podUID="8d53735c-521f-45e0-9653-84d72efd49e2" containerName="payload-processing" Oct 30 21:00:25.617855 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.617802 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d53735c-521f-45e0-9653-84d72efd49e2" containerName="payload-processing" Oct 30 21:00:25.618102 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.617860 4848 cpu_manager.go:414] "containerMap: removing container" podUID="21ccfd53-5388-4265-a017-22f7cfb377cb" containerName="revoke-keys" Oct 30 21:00:25.618102 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.617869 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ccfd53-5388-4265-a017-22f7cfb377cb" containerName="revoke-keys" Oct 30 21:00:25.618102 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.618052 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e85613-de9d-471e-b363-0eeb8c7972fe" containerName="payload-pre-processing" Oct 30 21:00:25.618102 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.618063 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72c5dba-3c91-48e6-90ea-b28ec0a23538" containerName="revoke-keys" Oct 30 21:00:25.618102 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.618073 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d53735c-521f-45e0-9653-84d72efd49e2" containerName="payload-processing" Oct 30 21:00:25.618102 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.618082 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ae86a2-a837-481f-8e1d-73d846addfbf" containerName="cleanup" Oct 30 21:00:25.618102 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.618100 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ccfd53-5388-4265-a017-22f7cfb377cb" containerName="revoke-keys" Oct 30 21:00:25.618312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.618114 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5fcaf4f-96eb-4f50-8de3-b5755dad797f" containerName="cleanup" Oct 30 21:00:25.625181 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.625162 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq" Oct 30 21:00:25.626008 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.625989 4848 scope.go:122] "RemoveContainer" containerID="3f50f3509d183e2604e8840370ca8ca3ce11ef494ca7fec4c9428ec159ef8081" Oct 30 21:00:25.626392 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:00:25.626329 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f50f3509d183e2604e8840370ca8ca3ce11ef494ca7fec4c9428ec159ef8081\": container with ID starting with 3f50f3509d183e2604e8840370ca8ca3ce11ef494ca7fec4c9428ec159ef8081 not found: ID does not exist" containerID="3f50f3509d183e2604e8840370ca8ca3ce11ef494ca7fec4c9428ec159ef8081" Oct 30 21:00:25.626640 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.626386 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f50f3509d183e2604e8840370ca8ca3ce11ef494ca7fec4c9428ec159ef8081"} err="failed to get container status \"3f50f3509d183e2604e8840370ca8ca3ce11ef494ca7fec4c9428ec159ef8081\": rpc error: code = NotFound desc = could not find container \"3f50f3509d183e2604e8840370ca8ca3ce11ef494ca7fec4c9428ec159ef8081\": container with ID starting with 3f50f3509d183e2604e8840370ca8ca3ce11ef494ca7fec4c9428ec159ef8081 not found: ID does not exist" Oct 30 21:00:25.626640 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.626431 4848 scope.go:122] "RemoveContainer" containerID="49d64ea267963f68ca859917eea640fd01b3eb5cf30134305c042fbbce828ed9" Oct 30 21:00:25.626640 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.626630 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq"] Oct 30 21:00:25.629913 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.629890 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7c7768\"" Oct 30 21:00:25.643104 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.642744 4848 scope.go:122] "RemoveContainer" containerID="49d64ea267963f68ca859917eea640fd01b3eb5cf30134305c042fbbce828ed9" Oct 30 21:00:25.643253 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:00:25.643144 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49d64ea267963f68ca859917eea640fd01b3eb5cf30134305c042fbbce828ed9\": container with ID starting with 49d64ea267963f68ca859917eea640fd01b3eb5cf30134305c042fbbce828ed9 not found: ID does not exist" containerID="49d64ea267963f68ca859917eea640fd01b3eb5cf30134305c042fbbce828ed9" Oct 30 21:00:25.643253 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.643180 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49d64ea267963f68ca859917eea640fd01b3eb5cf30134305c042fbbce828ed9"} err="failed to get container status \"49d64ea267963f68ca859917eea640fd01b3eb5cf30134305c042fbbce828ed9\": rpc error: code = NotFound desc = could not find container \"49d64ea267963f68ca859917eea640fd01b3eb5cf30134305c042fbbce828ed9\": container with ID starting with 49d64ea267963f68ca859917eea640fd01b3eb5cf30134305c042fbbce828ed9 not found: ID does not exist" Oct 30 21:00:25.647271 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.647245 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755"] Oct 30 21:00:25.652159 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.652137 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755" Oct 30 21:00:25.661662 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.661619 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7c7768-dockercfg-46ngc\"" Oct 30 21:00:25.661800 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.661682 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v"] Oct 30 21:00:25.666586 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.666561 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755"] Oct 30 21:00:25.669911 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.669889 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-7bc76854cf-nbd5v"] Oct 30 21:00:25.683344 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.683317 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf"] Oct 30 21:00:25.690794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.690739 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c7768-66557bc9ff-fstkf"] Oct 30 21:00:25.736307 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.736276 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5fd2739-f2e8-48d3-8d56-e15367a9c620-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq\" (UID: \"a5fd2739-f2e8-48d3-8d56-e15367a9c620\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq" Oct 30 21:00:25.736480 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.736333 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356460c8-2576-4249-8ba3-6efc7c78d124-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755\" (UID: \"356460c8-2576-4249-8ba3-6efc7c78d124\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755" Oct 30 21:00:25.736480 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.736443 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8497\" (UniqueName: \"kubernetes.io/projected/a5fd2739-f2e8-48d3-8d56-e15367a9c620-kube-api-access-j8497\") pod \"payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq\" (UID: \"a5fd2739-f2e8-48d3-8d56-e15367a9c620\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq" Oct 30 21:00:25.736578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.736561 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9m2g\" (UniqueName: \"kubernetes.io/projected/356460c8-2576-4249-8ba3-6efc7c78d124-kube-api-access-g9m2g\") pod \"payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755\" (UID: \"356460c8-2576-4249-8ba3-6efc7c78d124\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755" Oct 30 21:00:25.837677 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.837637 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9m2g\" (UniqueName: \"kubernetes.io/projected/356460c8-2576-4249-8ba3-6efc7c78d124-kube-api-access-g9m2g\") pod \"payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755\" (UID: \"356460c8-2576-4249-8ba3-6efc7c78d124\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755" Oct 30 21:00:25.841788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.838058 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5fd2739-f2e8-48d3-8d56-e15367a9c620-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq\" (UID: \"a5fd2739-f2e8-48d3-8d56-e15367a9c620\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq" Oct 30 21:00:25.856021 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.855986 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356460c8-2576-4249-8ba3-6efc7c78d124-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755\" (UID: \"356460c8-2576-4249-8ba3-6efc7c78d124\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755" Oct 30 21:00:25.857189 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.857162 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8497\" (UniqueName: \"kubernetes.io/projected/a5fd2739-f2e8-48d3-8d56-e15367a9c620-kube-api-access-j8497\") pod \"payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq\" (UID: \"a5fd2739-f2e8-48d3-8d56-e15367a9c620\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq" Oct 30 21:00:25.857448 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.838943 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5fd2739-f2e8-48d3-8d56-e15367a9c620-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq\" (UID: \"a5fd2739-f2e8-48d3-8d56-e15367a9c620\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq" Oct 30 21:00:25.857559 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.857062 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356460c8-2576-4249-8ba3-6efc7c78d124-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755\" (UID: \"356460c8-2576-4249-8ba3-6efc7c78d124\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755" Oct 30 21:00:25.858682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.858642 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9m2g\" (UniqueName: \"kubernetes.io/projected/356460c8-2576-4249-8ba3-6efc7c78d124-kube-api-access-g9m2g\") pod \"payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755\" (UID: \"356460c8-2576-4249-8ba3-6efc7c78d124\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755" Oct 30 21:00:25.899553 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.899474 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8497\" (UniqueName: \"kubernetes.io/projected/a5fd2739-f2e8-48d3-8d56-e15367a9c620-kube-api-access-j8497\") pod \"payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq\" (UID: \"a5fd2739-f2e8-48d3-8d56-e15367a9c620\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq" Oct 30 21:00:25.963349 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.962017 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq" Oct 30 21:00:25.986676 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:25.980998 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755" Oct 30 21:00:26.301967 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:26.301935 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq"] Oct 30 21:00:26.306309 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:00:26.306273 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5fd2739_f2e8_48d3_8d56_e15367a9c620.slice/crio-b79309491ec7f0b60844f92a5140ca9254f283dcb005b58961323c15d66e201b WatchSource:0}: Error finding container b79309491ec7f0b60844f92a5140ca9254f283dcb005b58961323c15d66e201b: Status 404 returned error can't find the container with id b79309491ec7f0b60844f92a5140ca9254f283dcb005b58961323c15d66e201b Oct 30 21:00:26.329047 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:00:26.329015 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356460c8_2576_4249_8ba3_6efc7c78d124.slice/crio-609742c1564df6b219535a822092426a7e091f0dca8a976544096d8ca98e616d WatchSource:0}: Error finding container 609742c1564df6b219535a822092426a7e091f0dca8a976544096d8ca98e616d: Status 404 returned error can't find the container with id 609742c1564df6b219535a822092426a7e091f0dca8a976544096d8ca98e616d Oct 30 21:00:26.332693 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:26.332668 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755"] Oct 30 21:00:26.628355 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:26.628249 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755" event={"ID":"356460c8-2576-4249-8ba3-6efc7c78d124","Type":"ContainerStarted","Data":"9e994a582be888497386789eb0c7da332706f94258db2f331d292a8c791bb1a4"} Oct 30 21:00:26.628524 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:26.628499 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755" Oct 30 21:00:26.628593 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:26.628547 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755" event={"ID":"356460c8-2576-4249-8ba3-6efc7c78d124","Type":"ContainerStarted","Data":"609742c1564df6b219535a822092426a7e091f0dca8a976544096d8ca98e616d"} Oct 30 21:00:26.631249 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:26.631214 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq" event={"ID":"a5fd2739-f2e8-48d3-8d56-e15367a9c620","Type":"ContainerStarted","Data":"fc0b95d610755d032a4807081b1a9ba38856ff721882e8c2af430202d603631c"} Oct 30 21:00:26.631418 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:26.631264 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq" event={"ID":"a5fd2739-f2e8-48d3-8d56-e15367a9c620","Type":"ContainerStarted","Data":"b79309491ec7f0b60844f92a5140ca9254f283dcb005b58961323c15d66e201b"} Oct 30 21:00:26.631985 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:26.631951 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq" Oct 30 21:00:26.650610 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:26.650264 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755" podStartSLOduration=1.6502465769999999 podStartE2EDuration="1.650246577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:26.64692394 +0000 UTC m=+1331.826820449" watchObservedRunningTime="2026-10-30 21:00:26.650246577 +0000 UTC m=+1331.830143154" Oct 30 21:00:26.667842 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:26.667758 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq" podStartSLOduration=1.667743164 podStartE2EDuration="1.667743164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:26.664237308 +0000 UTC m=+1331.844133816" watchObservedRunningTime="2026-10-30 21:00:26.667743164 +0000 UTC m=+1331.847639671" Oct 30 21:00:26.937406 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:26.937362 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54e85613-de9d-471e-b363-0eeb8c7972fe" path="/var/lib/kubelet/pods/54e85613-de9d-471e-b363-0eeb8c7972fe/volumes" Oct 30 21:00:26.937898 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:26.937879 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d53735c-521f-45e0-9653-84d72efd49e2" path="/var/lib/kubelet/pods/8d53735c-521f-45e0-9653-84d72efd49e2/volumes" Oct 30 21:00:29.661503 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:29.661419 4848 generic.go:356] "Generic (PLEG): container finished" podID="3c8ef112-9213-4f15-959c-63e09bb7faf5" containerID="15b5d23359bf9635b589a8cd2493ceec0ed416cbd845253702a20a9c66ca9c59" exitCode=0 Oct 30 21:00:29.662008 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:29.661495 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" event={"ID":"3c8ef112-9213-4f15-959c-63e09bb7faf5","Type":"ContainerDied","Data":"15b5d23359bf9635b589a8cd2493ceec0ed416cbd845253702a20a9c66ca9c59"} Oct 30 21:00:29.662008 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:29.661909 4848 scope.go:122] "RemoveContainer" containerID="15b5d23359bf9635b589a8cd2493ceec0ed416cbd845253702a20a9c66ca9c59" Oct 30 21:00:29.663245 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:29.663215 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:29.663334 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:29.663283 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:29.663334 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:29.663314 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:30.670291 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:30.670259 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" event={"ID":"3c8ef112-9213-4f15-959c-63e09bb7faf5","Type":"ContainerStarted","Data":"9b779f3d9283047cc8faa6fd5db7216caadf9dcf36a2c1a626dc19832445f120"} Oct 30 21:00:31.163269 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:31.163228 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 21:00:31.163639 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:31.163605 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 21:00:31.163831 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:31.163801 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" podUID="3c8ef112-9213-4f15-959c-63e09bb7faf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 21:00:32.174791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:32.167263 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 21:00:32.174791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:32.167370 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" podUID="3c8ef112-9213-4f15-959c-63e09bb7faf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 21:00:33.148791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.148199 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk"] Oct 30 21:00:33.196792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.178889 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 21:00:33.196792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.178945 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" podUID="3c8ef112-9213-4f15-959c-63e09bb7faf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 21:00:33.283791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.278435 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt"] Oct 30 21:00:33.298819 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.284147 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.374793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.361983 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp"] Oct 30 21:00:33.397706 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.397667 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk"] Oct 30 21:00:33.397910 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.397830 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.398004 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.397987 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.467353 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.467429 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.467464 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcm9s\" (UniqueName: \"kubernetes.io/projected/672d778a-896f-479e-877f-473372d7456c-kube-api-access-bcm9s\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.467501 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.467540 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/672d778a-896f-479e-877f-473372d7456c-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.467583 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.467622 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.467647 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.467686 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.467738 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.467802 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.467839 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.467864 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.467902 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.467931 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.467978 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.468018 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.468063 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.468093 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.468133 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkgrw\" (UniqueName: \"kubernetes.io/projected/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-kube-api-access-lkgrw\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.468157 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.468189 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.468219 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.468254 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5zd7\" (UniqueName: \"kubernetes.io/projected/087bab2d-59bc-4d7e-a526-2d84d6a2954f-kube-api-access-c5zd7\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.468280 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/672d778a-896f-479e-877f-473372d7456c-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.468323 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/672d778a-896f-479e-877f-473372d7456c-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.469796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.468355 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.536790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.531460 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt"] Oct 30 21:00:33.572789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.571801 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp"] Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.584839 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.584915 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.584953 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.584992 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkgrw\" (UniqueName: \"kubernetes.io/projected/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-kube-api-access-lkgrw\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585023 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585054 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585080 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585111 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zd7\" (UniqueName: \"kubernetes.io/projected/087bab2d-59bc-4d7e-a526-2d84d6a2954f-kube-api-access-c5zd7\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585134 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/672d778a-896f-479e-877f-473372d7456c-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585167 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/672d778a-896f-479e-877f-473372d7456c-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585196 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585266 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585330 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585361 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcm9s\" (UniqueName: \"kubernetes.io/projected/672d778a-896f-479e-877f-473372d7456c-kube-api-access-bcm9s\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585399 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585439 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/672d778a-896f-479e-877f-473372d7456c-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585486 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585532 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585557 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585601 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585668 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585720 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585759 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585804 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585842 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585872 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.586794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.585924 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.602516 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.587134 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.602516 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.587514 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.622801 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.603412 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.622801 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.603986 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.622801 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.604486 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:33.625819 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.625786 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.631057 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.630399 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.631057 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.630703 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.631057 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.630986 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-workload-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.636787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.636144 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/672d778a-896f-479e-877f-473372d7456c-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.639320 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.637362 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.639320 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.637630 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.663907 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.642431 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-data\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.663907 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.646477 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.663907 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.646748 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-credential-socket\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.663907 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.647034 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-workload-certs\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.668145 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.668113 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istiod-ca-cert\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.672809 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.671382 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.713792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.709229 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/672d778a-896f-479e-877f-473372d7456c-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.713792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.709797 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-token\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.714815 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.714679 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.725500 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.724551 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.725500 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.724811 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-istio-envoy\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.725500 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.725409 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5zd7\" (UniqueName: \"kubernetes.io/projected/087bab2d-59bc-4d7e-a526-2d84d6a2954f-kube-api-access-c5zd7\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.762173 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.762137 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/672d778a-896f-479e-877f-473372d7456c-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.763019 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.762995 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-podinfo\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:33.763726 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.763704 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcm9s\" (UniqueName: \"kubernetes.io/projected/672d778a-896f-479e-877f-473372d7456c-kube-api-access-bcm9s\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.764428 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.764403 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkgrw\" (UniqueName: \"kubernetes.io/projected/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-kube-api-access-lkgrw\") pod \"e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.768648 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.768629 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:33.776397 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.773244 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:33.783876 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:33.783854 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:34.141367 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.141320 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk"] Oct 30 21:00:34.142520 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:00:34.142490 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672d778a_896f_479e_877f_473372d7456c.slice/crio-6ce1f54b140df4e091263341e502196fa3d0c8f0f1c52262dae42250c16e3ee5 WatchSource:0}: Error finding container 6ce1f54b140df4e091263341e502196fa3d0c8f0f1c52262dae42250c16e3ee5: Status 404 returned error can't find the container with id 6ce1f54b140df4e091263341e502196fa3d0c8f0f1c52262dae42250c16e3ee5 Oct 30 21:00:34.146004 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.145968 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:34.146092 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.146034 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:34.146092 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.146066 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:34.154717 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.154684 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 21:00:34.154878 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.154809 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" podUID="3c8ef112-9213-4f15-959c-63e09bb7faf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 21:00:34.186760 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.186738 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt"] Oct 30 21:00:34.188676 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:00:34.188642 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod087bab2d_59bc_4d7e_a526_2d84d6a2954f.slice/crio-d18dcec1337513db296d7c75facbe89c72a3830254aef32bd998c091e985f708 WatchSource:0}: Error finding container d18dcec1337513db296d7c75facbe89c72a3830254aef32bd998c091e985f708: Status 404 returned error can't find the container with id d18dcec1337513db296d7c75facbe89c72a3830254aef32bd998c091e985f708 Oct 30 21:00:34.191607 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.191575 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:34.191700 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.191644 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:34.191700 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.191673 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:34.198690 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.198668 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp"] Oct 30 21:00:34.203007 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:00:34.202977 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7cbc43b_dad3_4ab6_ad12_e03522a725b0.slice/crio-5225e7cac38703db8b658fac39351a2a12029e3fecb426f7e4e8aba7486207b3 WatchSource:0}: Error finding container 5225e7cac38703db8b658fac39351a2a12029e3fecb426f7e4e8aba7486207b3: Status 404 returned error can't find the container with id 5225e7cac38703db8b658fac39351a2a12029e3fecb426f7e4e8aba7486207b3 Oct 30 21:00:34.232812 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.227254 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:34.232812 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.227342 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:34.232812 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.227384 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:34.785662 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.785623 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" event={"ID":"087bab2d-59bc-4d7e-a526-2d84d6a2954f","Type":"ContainerStarted","Data":"ff6c870679bf367d52efb602a2ae70bbc8a9b8e5f3d529d953b6019b9541e344"} Oct 30 21:00:34.785955 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.785932 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" event={"ID":"087bab2d-59bc-4d7e-a526-2d84d6a2954f","Type":"ContainerStarted","Data":"d18dcec1337513db296d7c75facbe89c72a3830254aef32bd998c091e985f708"} Oct 30 21:00:34.788663 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.788629 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:34.794784 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.794726 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" event={"ID":"f7cbc43b-dad3-4ab6-ad12-e03522a725b0","Type":"ContainerStarted","Data":"a9b066e264757d53ee38596b8c4892f594b83d461de429ae76a63c632c11b340"} Oct 30 21:00:34.795049 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.795026 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" event={"ID":"f7cbc43b-dad3-4ab6-ad12-e03522a725b0","Type":"ContainerStarted","Data":"5225e7cac38703db8b658fac39351a2a12029e3fecb426f7e4e8aba7486207b3"} Oct 30 21:00:34.798981 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.798409 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" event={"ID":"672d778a-896f-479e-877f-473372d7456c","Type":"ContainerStarted","Data":"3710606c84f011220dc73ec01e856f2084e59d2cc086aac1518f2854ee7c3a1c"} Oct 30 21:00:34.799166 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.799145 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" event={"ID":"672d778a-896f-479e-877f-473372d7456c","Type":"ContainerStarted","Data":"6ce1f54b140df4e091263341e502196fa3d0c8f0f1c52262dae42250c16e3ee5"} Oct 30 21:00:34.861920 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.859492 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" podStartSLOduration=2.85947228 podStartE2EDuration="2.85947228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:34.826513809 +0000 UTC m=+1340.006410319" watchObservedRunningTime="2026-10-30 21:00:34.85947228 +0000 UTC m=+1340.039368790" Oct 30 21:00:34.868482 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.863357 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" podStartSLOduration=2.863341406 podStartE2EDuration="2.863341406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:34.8594129 +0000 UTC m=+1340.039309408" watchObservedRunningTime="2026-10-30 21:00:34.863341406 +0000 UTC m=+1340.043237907" Oct 30 21:00:34.893955 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:34.893907 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" podStartSLOduration=2.893893404 podStartE2EDuration="2.893893404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:34.891468442 +0000 UTC m=+1340.071364966" watchObservedRunningTime="2026-10-30 21:00:34.893893404 +0000 UTC m=+1340.073789987" Oct 30 21:00:35.160269 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:35.160174 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 21:00:35.160269 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:35.160241 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" podUID="3c8ef112-9213-4f15-959c-63e09bb7faf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 21:00:35.805793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:35.804831 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:35.827426 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:35.827389 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:00:35.828821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:35.828786 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" podUID="f7cbc43b-dad3-4ab6-ad12-e03522a725b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 21:00:35.829254 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:35.827691 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:35.843996 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:35.843960 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 21:00:35.844280 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:35.844253 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" podUID="672d778a-896f-479e-877f-473372d7456c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 21:00:35.925808 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:35.871260 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 21:00:35.925808 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:35.871316 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" podUID="087bab2d-59bc-4d7e-a526-2d84d6a2954f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 21:00:36.278692 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:36.278657 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 21:00:36.279027 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:36.278987 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" podUID="3c8ef112-9213-4f15-959c-63e09bb7faf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 21:00:36.577600 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:36.576911 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:36.786792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:36.774562 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 21:00:36.786792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:36.774630 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" podUID="672d778a-896f-479e-877f-473372d7456c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 21:00:36.786792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:36.774727 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:00:36.786792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:36.774750 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" podUID="f7cbc43b-dad3-4ab6-ad12-e03522a725b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 21:00:36.791993 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:36.791961 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 21:00:36.792248 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:36.792219 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" podUID="087bab2d-59bc-4d7e-a526-2d84d6a2954f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 21:00:37.093570 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:37.093526 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:37.188656 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:37.188610 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 21:00:37.189525 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:37.189505 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 21:00:37.190872 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:37.190845 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 21:00:37.648791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:37.648625 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755" Oct 30 21:00:37.779805 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:37.771190 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 21:00:37.779805 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:37.771251 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" podUID="672d778a-896f-479e-877f-473372d7456c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 21:00:37.791848 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:37.784751 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 21:00:37.791848 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:37.784823 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" podUID="087bab2d-59bc-4d7e-a526-2d84d6a2954f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 21:00:37.791848 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:37.784900 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:00:37.791848 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:37.784920 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" podUID="f7cbc43b-dad3-4ab6-ad12-e03522a725b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 21:00:38.685415 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:38.684009 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq" Oct 30 21:00:38.797402 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:38.796913 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:00:38.797402 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:38.796973 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" podUID="f7cbc43b-dad3-4ab6-ad12-e03522a725b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 21:00:38.797402 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:38.796915 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 21:00:38.797402 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:38.797046 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" podUID="087bab2d-59bc-4d7e-a526-2d84d6a2954f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 21:00:38.797402 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:38.797112 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 21:00:38.797402 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:38.797136 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" podUID="672d778a-896f-479e-877f-473372d7456c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 21:00:39.785082 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:39.785044 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 21:00:39.785675 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:39.785644 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" podUID="672d778a-896f-479e-877f-473372d7456c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 21:00:39.785905 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:39.785144 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 21:00:39.786030 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:39.786011 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" podUID="087bab2d-59bc-4d7e-a526-2d84d6a2954f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 21:00:39.786129 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:39.785175 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:00:39.786217 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:39.786202 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" podUID="f7cbc43b-dad3-4ab6-ad12-e03522a725b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 21:00:40.778791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:40.778509 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 21:00:40.778791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:40.778571 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" podUID="672d778a-896f-479e-877f-473372d7456c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 21:00:40.779071 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:40.779001 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:00:40.779071 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:40.779037 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" podUID="f7cbc43b-dad3-4ab6-ad12-e03522a725b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 21:00:40.796358 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:40.788669 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 21:00:40.796358 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:40.788721 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" podUID="087bab2d-59bc-4d7e-a526-2d84d6a2954f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 21:00:41.786914 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:41.786867 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 21:00:41.787129 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:41.786959 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" podUID="f7cbc43b-dad3-4ab6-ad12-e03522a725b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 21:00:41.787210 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:41.787145 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 21:00:41.787210 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:41.787195 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" podUID="087bab2d-59bc-4d7e-a526-2d84d6a2954f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 21:00:41.787341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:41.786892 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 21:00:41.787480 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:41.787454 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" podUID="672d778a-896f-479e-877f-473372d7456c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 21:00:42.341218 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:42.341179 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f"] Oct 30 21:00:42.342122 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:42.342085 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" podUID="e80f4869-c21f-4f08-b997-ac9c56c23619" containerName="maas-api" containerID="cri-o://1d812e26831edce3742eb53279c186269f880050b43c07b899cdd0f4b63ec083" gracePeriod=30 Oct 30 21:00:43.257178 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.257143 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" Oct 30 21:00:43.354383 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.354285 4848 generic.go:356] "Generic (PLEG): container finished" podID="e80f4869-c21f-4f08-b997-ac9c56c23619" containerID="1d812e26831edce3742eb53279c186269f880050b43c07b899cdd0f4b63ec083" exitCode=0 Oct 30 21:00:43.355113 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.355086 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" event={"ID":"e80f4869-c21f-4f08-b997-ac9c56c23619","Type":"ContainerDied","Data":"1d812e26831edce3742eb53279c186269f880050b43c07b899cdd0f4b63ec083"} Oct 30 21:00:43.355332 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.355298 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" event={"ID":"e80f4869-c21f-4f08-b997-ac9c56c23619","Type":"ContainerDied","Data":"f5fcd22882d1774db7681b5f829b761d53e3324dbae2da3956cbc0f949d8f039"} Oct 30 21:00:43.355733 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.355483 4848 scope.go:122] "RemoveContainer" containerID="1d812e26831edce3742eb53279c186269f880050b43c07b899cdd0f4b63ec083" Oct 30 21:00:43.356035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.355566 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f" Oct 30 21:00:43.372330 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.372044 4848 scope.go:122] "RemoveContainer" containerID="1d812e26831edce3742eb53279c186269f880050b43c07b899cdd0f4b63ec083" Oct 30 21:00:43.372654 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:00:43.372555 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d812e26831edce3742eb53279c186269f880050b43c07b899cdd0f4b63ec083\": container with ID starting with 1d812e26831edce3742eb53279c186269f880050b43c07b899cdd0f4b63ec083 not found: ID does not exist" containerID="1d812e26831edce3742eb53279c186269f880050b43c07b899cdd0f4b63ec083" Oct 30 21:00:43.372654 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.372593 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d812e26831edce3742eb53279c186269f880050b43c07b899cdd0f4b63ec083"} err="failed to get container status \"1d812e26831edce3742eb53279c186269f880050b43c07b899cdd0f4b63ec083\": rpc error: code = NotFound desc = could not find container \"1d812e26831edce3742eb53279c186269f880050b43c07b899cdd0f4b63ec083\": container with ID starting with 1d812e26831edce3742eb53279c186269f880050b43c07b899cdd0f4b63ec083 not found: ID does not exist" Oct 30 21:00:43.419456 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.413559 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e80f4869-c21f-4f08-b997-ac9c56c23619-kube-api-access-c46kw\" (UniqueName: \"kubernetes.io/projected/e80f4869-c21f-4f08-b997-ac9c56c23619-kube-api-access-c46kw\") pod \"e80f4869-c21f-4f08-b997-ac9c56c23619\" (UID: \"e80f4869-c21f-4f08-b997-ac9c56c23619\") " Oct 30 21:00:43.419945 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.419913 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e80f4869-c21f-4f08-b997-ac9c56c23619-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80f4869-c21f-4f08-b997-ac9c56c23619-openshift-service-ca\") pod \"e80f4869-c21f-4f08-b997-ac9c56c23619\" (UID: \"e80f4869-c21f-4f08-b997-ac9c56c23619\") " Oct 30 21:00:43.424384 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.420154 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e80f4869-c21f-4f08-b997-ac9c56c23619-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e80f4869-c21f-4f08-b997-ac9c56c23619-metrics-certs\") pod \"e80f4869-c21f-4f08-b997-ac9c56c23619\" (UID: \"e80f4869-c21f-4f08-b997-ac9c56c23619\") " Oct 30 21:00:43.424726 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.424704 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e80f4869-c21f-4f08-b997-ac9c56c23619-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e80f4869-c21f-4f08-b997-ac9c56c23619-maas-api-tls\") pod \"e80f4869-c21f-4f08-b997-ac9c56c23619\" (UID: \"e80f4869-c21f-4f08-b997-ac9c56c23619\") " Oct 30 21:00:43.426660 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.426409 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e80f4869-c21f-4f08-b997-ac9c56c23619-openshift-service-ca" pod "e80f4869-c21f-4f08-b997-ac9c56c23619" (UID: "e80f4869-c21f-4f08-b997-ac9c56c23619"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:43.452409 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.452353 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e80f4869-c21f-4f08-b997-ac9c56c23619-metrics-certs" pod "e80f4869-c21f-4f08-b997-ac9c56c23619" (UID: "e80f4869-c21f-4f08-b997-ac9c56c23619"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:43.462236 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.462179 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e80f4869-c21f-4f08-b997-ac9c56c23619-maas-api-tls" pod "e80f4869-c21f-4f08-b997-ac9c56c23619" (UID: "e80f4869-c21f-4f08-b997-ac9c56c23619"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:43.471257 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.471206 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e80f4869-c21f-4f08-b997-ac9c56c23619-kube-api-access-c46kw" pod "e80f4869-c21f-4f08-b997-ac9c56c23619" (UID: "e80f4869-c21f-4f08-b997-ac9c56c23619"). InnerVolumeSpecName "kube-api-access-c46kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:43.542792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.539866 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e80f4869-c21f-4f08-b997-ac9c56c23619-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.542792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.539905 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c46kw\" (UniqueName: \"kubernetes.io/projected/e80f4869-c21f-4f08-b997-ac9c56c23619-kube-api-access-c46kw\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.542792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.539921 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e80f4869-c21f-4f08-b997-ac9c56c23619-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.542792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.539936 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e80f4869-c21f-4f08-b997-ac9c56c23619-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:43.561789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.551089 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:43.561789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.552626 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:43.581796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.578184 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:00:43.826650 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.825899 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:00:43.826650 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.825983 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" podUID="672d778a-896f-479e-877f-473372d7456c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:43.837939 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.837902 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:00:43.839988 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:43.838890 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" podUID="f7cbc43b-dad3-4ab6-ad12-e03522a725b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:44.038127 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.038040 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f"] Oct 30 21:00:44.203788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.198838 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-955eb7-56446fb849-7kt7f"] Oct 30 21:00:44.270149 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.251309 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:44.270149 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.252641 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:44.277675 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.276872 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:00:44.320843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.309010 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z"] Oct 30 21:00:44.322074 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.322040 4848 cpu_manager.go:414] "containerMap: removing container" podUID="e80f4869-c21f-4f08-b997-ac9c56c23619" containerName="maas-api" Oct 30 21:00:44.322249 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.322234 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e80f4869-c21f-4f08-b997-ac9c56c23619" containerName="maas-api" Oct 30 21:00:44.322604 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.322588 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="e80f4869-c21f-4f08-b997-ac9c56c23619" containerName="maas-api" Oct 30 21:00:44.430786 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.427243 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z"] Oct 30 21:00:44.430786 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.427359 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:44.482791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.442302 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b46fb7\"/\"ipp-route-b46fb7-kserve-self-signed-certs\"" Oct 30 21:00:44.482791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.443026 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b46fb7\"/\"kube-root-ca.crt\"" Oct 30 21:00:44.482791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.443106 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b46fb7\"/\"openshift-service-ca.crt\"" Oct 30 21:00:44.482791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.443203 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b46fb7\"/\"default-dockercfg-dhpr8\"" Oct 30 21:00:44.571731 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.571622 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-model-cache\") pod \"ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:44.575508 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.572059 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-tmp-dir\") pod \"ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:44.575508 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.572217 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-home\") pod \"ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:44.575508 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.572250 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14ba0389-1004-4f23-8560-cf6fe6f597dd-tls-certs\") pod \"ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:44.575508 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.572305 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-dshm\") pod \"ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:44.575508 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.572393 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmv2n\" (UniqueName: \"kubernetes.io/projected/14ba0389-1004-4f23-8560-cf6fe6f597dd-kube-api-access-qmv2n\") pod \"ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:44.575508 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.572646 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5"] Oct 30 21:00:44.672790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.655115 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.693789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.683223 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.718963 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.718933 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmv2n\" (UniqueName: \"kubernetes.io/projected/14ba0389-1004-4f23-8560-cf6fe6f597dd-kube-api-access-qmv2n\") pod \"ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:44.719378 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.719356 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.719800 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.719754 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvr4c\" (UniqueName: \"kubernetes.io/projected/9835b513-5c93-42f0-90e4-bfc33b2b8a43-kube-api-access-bvr4c\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.719992 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.719966 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.720665 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.720293 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-model-cache\") pod \"ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:44.720921 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.720899 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-tmp-dir\") pod \"ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:44.721730 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.721708 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.722324 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.722297 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.722475 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.722457 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.722637 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.722620 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.752803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.752658 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-model-cache\") pod \"ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:44.752803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.718393 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm"] Oct 30 21:00:44.787232 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.787165 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-tmp-dir\") pod \"ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:44.809188 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.809152 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-home\") pod \"ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:44.810939 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.722758 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-home\") pod \"ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:44.811234 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.811212 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14ba0389-1004-4f23-8560-cf6fe6f597dd-tls-certs\") pod \"ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:44.811385 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.811364 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.811548 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.811531 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-dshm\") pod \"ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:44.851472 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.828071 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:00:44.851472 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.828136 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" podUID="672d778a-896f-479e-877f-473372d7456c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:44.851472 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.844098 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-dshm\") pod \"ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:44.886474 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.886440 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14ba0389-1004-4f23-8560-cf6fe6f597dd-tls-certs\") pod \"ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:44.932399 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.932360 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.932683 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.932659 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvr4c\" (UniqueName: \"kubernetes.io/projected/9835b513-5c93-42f0-90e4-bfc33b2b8a43-kube-api-access-bvr4c\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.932997 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.932977 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.933275 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.933254 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.933473 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.933455 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.933598 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.933583 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.933838 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.933817 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.933999 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.933980 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.934180 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.934162 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.961957 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.961931 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:44.972711 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.972665 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.975584 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.975556 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.976334 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.976285 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.976784 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.976746 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:44.977679 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:44.977639 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:45.014412 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.014382 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:45.039185 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.039155 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:45.118743 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.118661 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvr4c\" (UniqueName: \"kubernetes.io/projected/9835b513-5c93-42f0-90e4-bfc33b2b8a43-kube-api-access-bvr4c\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:45.144954 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.144917 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmv2n\" (UniqueName: \"kubernetes.io/projected/14ba0389-1004-4f23-8560-cf6fe6f597dd-kube-api-access-qmv2n\") pod \"ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:45.154839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.149520 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw2wv\" (UniqueName: \"kubernetes.io/projected/9f136686-02a0-4906-a897-9d8a7c1d748d-kube-api-access-jw2wv\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.154839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.149665 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.154839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.149701 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.154839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.149793 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.154839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.149843 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.154839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.149914 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f136686-02a0-4906-a897-9d8a7c1d748d-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.154839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.149985 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.154839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.150116 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.154839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.150231 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.169795 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.156943 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:45.268843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.255501 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e80f4869-c21f-4f08-b997-ac9c56c23619" path="/var/lib/kubelet/pods/e80f4869-c21f-4f08-b997-ac9c56c23619/volumes" Oct 30 21:00:45.268843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.256201 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5"] Oct 30 21:00:45.268843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.256264 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm"] Oct 30 21:00:45.272720 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.272684 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.301386 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.301351 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.301632 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.274342 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:45.301852 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.301833 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f136686-02a0-4906-a897-9d8a7c1d748d-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.302187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.302168 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.302459 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.302440 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.302685 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.302665 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.306912 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.306888 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw2wv\" (UniqueName: \"kubernetes.io/projected/9f136686-02a0-4906-a897-9d8a7c1d748d-kube-api-access-jw2wv\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.307190 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.307171 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.307324 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.307308 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.307807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.307787 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-workload-certs\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.336421 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.303087 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-workload-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.336421 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.273518 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-credential-socket\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.336421 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.334323 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-data\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.336421 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.336108 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f136686-02a0-4906-a897-9d8a7c1d748d-istiod-ca-cert\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.380856 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.363295 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-podinfo\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.395826 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.385951 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:45.498857 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.498821 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw2wv\" (UniqueName: \"kubernetes.io/projected/9f136686-02a0-4906-a897-9d8a7c1d748d-kube-api-access-jw2wv\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.515821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.499961 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-token\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.544798 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.537190 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-envoy\") pod \"e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.612250 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.612215 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:45.951843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.931249 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:00:45.951843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:45.931317 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" podUID="672d778a-896f-479e-877f-473372d7456c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:46.283683 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:46.282898 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 21:00:46.301004 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:46.300957 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:46.598269 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:46.593865 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:46.957055 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:46.938111 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:47.001502 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:46.978354 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:00:47.001502 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:46.978416 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" podUID="672d778a-896f-479e-877f-473372d7456c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:47.041879 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:47.040851 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 21:00:47.367841 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:47.309916 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:47.386991 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:47.386946 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:00:47.508793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:47.506183 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 21:00:47.508793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:47.506248 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 21:00:47.546242 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:47.546212 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 21:00:47.614446 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:47.614366 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5"] Oct 30 21:00:47.621037 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:47.620259 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 21:00:47.635484 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:47.635115 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:47.635484 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:47.635203 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:47.635484 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:47.635246 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:47.717878 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:47.717845 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:47.719537 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:47.719518 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:47.720746 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:47.720729 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:00:47.763096 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:47.762981 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm"] Oct 30 21:00:47.826084 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:47.825651 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:47.826084 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:47.825818 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:47.826084 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:47.825916 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:47.897865 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:47.897830 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z"] Oct 30 21:00:48.069181 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:48.069144 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" event={"ID":"9f136686-02a0-4906-a897-9d8a7c1d748d","Type":"ContainerStarted","Data":"1438d1f7121f547e0bcf4e39b11b7b0bbedf58bd16bb2bcc59af4d64ba7e6df6"} Oct 30 21:00:48.070482 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:48.070457 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" event={"ID":"9f136686-02a0-4906-a897-9d8a7c1d748d","Type":"ContainerStarted","Data":"65132b009c5e8d1eb12d91ea5fdf6510ae30234c225ce0195ac91543ece5c4d7"} Oct 30 21:00:48.072705 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:48.072680 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" event={"ID":"9835b513-5c93-42f0-90e4-bfc33b2b8a43","Type":"ContainerStarted","Data":"08c3d64e0a6b713d139b76c403419da5815af0685b9413f463352b7e638b92f1"} Oct 30 21:00:48.074732 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:48.074713 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" event={"ID":"9835b513-5c93-42f0-90e4-bfc33b2b8a43","Type":"ContainerStarted","Data":"6a7178261bc063bcf7e9cee0e2d982b90aeb837f040c466a84110bed27ec4b69"} Oct 30 21:00:48.077933 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:48.077912 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" event={"ID":"14ba0389-1004-4f23-8560-cf6fe6f597dd","Type":"ContainerStarted","Data":"062aaa1f6d33641289152a8dfab16f311c6fb51dd038cee4ba4d20ec1b6cc91b"} Oct 30 21:00:48.078125 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:48.078109 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" event={"ID":"14ba0389-1004-4f23-8560-cf6fe6f597dd","Type":"ContainerStarted","Data":"381c85b9c4487f139b49dd4fe90a93aaf85cb565b86f92e485ae9ed4747b752c"} Oct 30 21:00:48.105548 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:48.105487 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" podStartSLOduration=4.105466452 podStartE2EDuration="4.105466452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:48.102545112 +0000 UTC m=+1353.282441623" watchObservedRunningTime="2026-10-30 21:00:48.105466452 +0000 UTC m=+1353.285362956" Oct 30 21:00:48.145722 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:48.145614 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" podStartSLOduration=4.145593895 podStartE2EDuration="4.145593895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:48.142188757 +0000 UTC m=+1353.322085266" watchObservedRunningTime="2026-10-30 21:00:48.145593895 +0000 UTC m=+1353.325490402" Oct 30 21:00:48.175675 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:48.174923 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" podStartSLOduration=5.174901135 podStartE2EDuration="5.174901135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:48.169744893 +0000 UTC m=+1353.349641400" watchObservedRunningTime="2026-10-30 21:00:48.174901135 +0000 UTC m=+1353.354797644" Oct 30 21:00:48.276977 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:48.276938 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:48.277146 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:48.277063 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 21:00:48.277235 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:48.277107 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" podUID="9835b513-5c93-42f0-90e4-bfc33b2b8a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 21:00:48.400091 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:48.400008 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:48.625712 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:48.622745 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:49.278793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:49.276478 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 21:00:49.278793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:49.276540 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" podUID="9835b513-5c93-42f0-90e4-bfc33b2b8a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 21:00:49.616852 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:49.616672 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 21:00:49.616852 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:49.616741 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" podUID="9f136686-02a0-4906-a897-9d8a7c1d748d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 21:00:50.275792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:50.275117 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 21:00:50.275792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:50.275190 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" podUID="9835b513-5c93-42f0-90e4-bfc33b2b8a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 21:00:50.617291 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:50.617191 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 21:00:50.617575 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:50.617543 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" podUID="9f136686-02a0-4906-a897-9d8a7c1d748d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 21:00:50.887051 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:50.886973 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm"] Oct 30 21:00:50.887452 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:50.887422 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" podUID="8fef8496-2113-44ff-9694-038139e4b194" containerName="istio-proxy" containerID="cri-o://40686b69066115ffd9c0019177fdfc0cb5654d8944bd3b7657fba97a5c2d31e5" gracePeriod=30 Oct 30 21:00:51.290977 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:51.290935 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 21:00:51.291710 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:51.291677 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" podUID="9835b513-5c93-42f0-90e4-bfc33b2b8a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 21:00:51.653420 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:51.652828 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 21:00:51.653420 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:51.652892 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" podUID="9f136686-02a0-4906-a897-9d8a7c1d748d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 21:00:52.275553 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:52.275514 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 21:00:52.275752 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:52.275572 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" podUID="9835b513-5c93-42f0-90e4-bfc33b2b8a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 21:00:52.565834 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:00:52.565715 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:00:52.629811 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:52.622921 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 21:00:52.629811 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:52.622979 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" podUID="9f136686-02a0-4906-a897-9d8a7c1d748d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 21:00:53.275343 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:53.275309 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 21:00:53.275536 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:53.275364 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" podUID="9835b513-5c93-42f0-90e4-bfc33b2b8a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 21:00:53.613498 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:53.613388 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 21:00:53.618301 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:53.618259 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" podUID="9f136686-02a0-4906-a897-9d8a7c1d748d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 21:00:54.275783 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:54.275737 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 21:00:54.275974 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:54.275801 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" podUID="9835b513-5c93-42f0-90e4-bfc33b2b8a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 21:00:54.613690 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:54.613584 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 30 21:00:54.613690 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:54.613648 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" podUID="9f136686-02a0-4906-a897-9d8a7c1d748d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 30 21:00:55.032356 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.032316 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p"] Oct 30 21:00:55.036442 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.036416 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.047176 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.047151 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg"] Oct 30 21:00:55.052581 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.052557 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.062665 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.062640 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p"] Oct 30 21:00:55.104880 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.101601 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg"] Oct 30 21:00:55.114701 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.114665 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.115128 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.115103 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.115312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.115293 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.115414 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.115351 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.115414 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.115391 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6vq9\" (UniqueName: \"kubernetes.io/projected/cf35d310-f121-41d1-844c-e2bef5d31d15-kube-api-access-z6vq9\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.115512 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.115417 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vn2z\" (UniqueName: \"kubernetes.io/projected/91bae7a3-eafb-4ec3-976b-d885576ecca1-kube-api-access-7vn2z\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.115512 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.115444 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.115646 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.115617 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.115715 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.115677 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.115759 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.115743 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.115759 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.115796 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.115957 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.115932 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.116035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.115973 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.116035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.116003 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91bae7a3-eafb-4ec3-976b-d885576ecca1-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.116035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.116029 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf35d310-f121-41d1-844c-e2bef5d31d15-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.116167 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.116139 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.116235 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.116215 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.116295 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.116275 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.218152 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.218113 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.218356 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.218167 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.218356 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.218232 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.218356 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.218290 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.218539 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.218379 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.218539 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.218417 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.218539 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.218476 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.218539 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.218521 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6vq9\" (UniqueName: \"kubernetes.io/projected/cf35d310-f121-41d1-844c-e2bef5d31d15-kube-api-access-z6vq9\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.218755 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.218553 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vn2z\" (UniqueName: \"kubernetes.io/projected/91bae7a3-eafb-4ec3-976b-d885576ecca1-kube-api-access-7vn2z\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.218755 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.218582 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.218755 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.218599 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.218755 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.218635 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.218755 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.218680 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.218755 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.218730 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.219129 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.218789 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.219129 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.218869 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.219129 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.218904 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.219129 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.218933 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91bae7a3-eafb-4ec3-976b-d885576ecca1-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.219129 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.218962 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf35d310-f121-41d1-844c-e2bef5d31d15-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.219129 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.218998 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.219601 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.219580 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-workload-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.220087 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.220065 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.220205 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.220184 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf35d310-f121-41d1-844c-e2bef5d31d15-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.220294 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.220137 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-workload-certs\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.220409 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.220381 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-credential-socket\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.220686 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.220661 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.220686 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.220661 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-data\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.221673 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.221398 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91bae7a3-eafb-4ec3-976b-d885576ecca1-istiod-ca-cert\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.222846 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.221592 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.223011 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.221630 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.223528 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.223501 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-podinfo\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.224603 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.224577 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-envoy\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.231162 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.231131 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.232066 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.232041 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vn2z\" (UniqueName: \"kubernetes.io/projected/91bae7a3-eafb-4ec3-976b-d885576ecca1-kube-api-access-7vn2z\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.233497 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.233452 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-token\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.235896 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.235867 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6vq9\" (UniqueName: \"kubernetes.io/projected/cf35d310-f121-41d1-844c-e2bef5d31d15-kube-api-access-z6vq9\") pod \"e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.275059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.275024 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body= Oct 30 21:00:55.275260 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.275105 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" podUID="9835b513-5c93-42f0-90e4-bfc33b2b8a43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" Oct 30 21:00:55.355436 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.355345 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:55.383790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.383736 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:55.401117 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.401086 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:55.428541 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.421957 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:55.830428 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.830397 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg"] Oct 30 21:00:55.831409 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.831372 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:55.831525 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.831453 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:55.831525 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.831493 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:55.883207 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.881416 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p"] Oct 30 21:00:55.887220 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:00:55.887188 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf35d310_f121_41d1_844c_e2bef5d31d15.slice/crio-9e475baa125d30b33eb65e5298866650955e0e47993ddba83312d1dc90e6fb7d WatchSource:0}: Error finding container 9e475baa125d30b33eb65e5298866650955e0e47993ddba83312d1dc90e6fb7d: Status 404 returned error can't find the container with id 9e475baa125d30b33eb65e5298866650955e0e47993ddba83312d1dc90e6fb7d Oct 30 21:00:55.890555 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.890523 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:55.890689 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.890589 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:55.890689 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:55.890618 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:00:56.265744 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.265717 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:56.268179 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.267235 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:56.269421 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.269371 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:00:56.293287 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.291658 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:56.293287 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.292869 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:56.295201 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.295178 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:00:56.297960 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.297934 4848 generic.go:356] "Generic (PLEG): container finished" podID="8fef8496-2113-44ff-9694-038139e4b194" containerID="40686b69066115ffd9c0019177fdfc0cb5654d8944bd3b7657fba97a5c2d31e5" exitCode=0 Oct 30 21:00:56.298075 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.298035 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" event={"ID":"8fef8496-2113-44ff-9694-038139e4b194","Type":"ContainerDied","Data":"40686b69066115ffd9c0019177fdfc0cb5654d8944bd3b7657fba97a5c2d31e5"} Oct 30 21:00:56.298146 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.298077 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" event={"ID":"8fef8496-2113-44ff-9694-038139e4b194","Type":"ContainerDied","Data":"a91b9a84d66beda3f5ce7e3f907d3ec76d9676577849312b4550912e74958cec"} Oct 30 21:00:56.298146 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.298101 4848 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a91b9a84d66beda3f5ce7e3f907d3ec76d9676577849312b4550912e74958cec" Oct 30 21:00:56.298146 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.298118 4848 scope.go:122] "RemoveContainer" containerID="fd6a60993c23f9881c200d1afe9b1f5c92930b5fb5b92a8cd853a88518ef0d85" Oct 30 21:00:56.300651 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.300629 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" event={"ID":"91bae7a3-eafb-4ec3-976b-d885576ecca1","Type":"ContainerStarted","Data":"24a534dbf554343df6b7fe2945cf6d7273c268a3a4e4887c7db28cd6f23fdd18"} Oct 30 21:00:56.300834 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.300802 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" event={"ID":"91bae7a3-eafb-4ec3-976b-d885576ecca1","Type":"ContainerStarted","Data":"525152342a242a7807905de8efeffc5df8d777d1096a93e6028aab448dd1381a"} Oct 30 21:00:56.305513 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.305497 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 21:00:56.315709 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.315681 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" event={"ID":"cf35d310-f121-41d1-844c-e2bef5d31d15","Type":"ContainerStarted","Data":"3247c5dd4fde2db7467047901b3ddb58bfe08d0fe715033b69cae9ef2c516f17"} Oct 30 21:00:56.315908 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.315893 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" event={"ID":"cf35d310-f121-41d1-844c-e2bef5d31d15","Type":"ContainerStarted","Data":"9e475baa125d30b33eb65e5298866650955e0e47993ddba83312d1dc90e6fb7d"} Oct 30 21:00:56.335915 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.335884 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:00:56.361444 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.361411 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:00:56.394515 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.394484 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:00:56.394828 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.394799 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-istio-envoy\") pod \"8fef8496-2113-44ff-9694-038139e4b194\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " Oct 30 21:00:56.394928 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.394911 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fef8496-2113-44ff-9694-038139e4b194-kube-api-access-rtb7h\" (UniqueName: \"kubernetes.io/projected/8fef8496-2113-44ff-9694-038139e4b194-kube-api-access-rtb7h\") pod \"8fef8496-2113-44ff-9694-038139e4b194\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " Oct 30 21:00:56.394990 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.394981 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fef8496-2113-44ff-9694-038139e4b194-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fef8496-2113-44ff-9694-038139e4b194-istiod-ca-cert\") pod \"8fef8496-2113-44ff-9694-038139e4b194\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " Oct 30 21:00:56.395070 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.395049 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fef8496-2113-44ff-9694-038139e4b194-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fef8496-2113-44ff-9694-038139e4b194-istio-podinfo\") pod \"8fef8496-2113-44ff-9694-038139e4b194\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " Oct 30 21:00:56.395154 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.395138 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-istio-data\") pod \"8fef8496-2113-44ff-9694-038139e4b194\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " Oct 30 21:00:56.395215 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.395180 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-workload-certs\") pod \"8fef8496-2113-44ff-9694-038139e4b194\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " Oct 30 21:00:56.395294 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.395273 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-workload-socket\") pod \"8fef8496-2113-44ff-9694-038139e4b194\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " Oct 30 21:00:56.395351 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.395330 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fef8496-2113-44ff-9694-038139e4b194-istio-token\" (UniqueName: \"kubernetes.io/projected/8fef8496-2113-44ff-9694-038139e4b194-istio-token\") pod \"8fef8496-2113-44ff-9694-038139e4b194\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " Oct 30 21:00:56.395428 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.395412 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-credential-socket\") pod \"8fef8496-2113-44ff-9694-038139e4b194\" (UID: \"8fef8496-2113-44ff-9694-038139e4b194\") " Oct 30 21:00:56.397347 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.397319 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-workload-certs" pod "8fef8496-2113-44ff-9694-038139e4b194" (UID: "8fef8496-2113-44ff-9694-038139e4b194"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:56.399994 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.399828 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-workload-socket" pod "8fef8496-2113-44ff-9694-038139e4b194" (UID: "8fef8496-2113-44ff-9694-038139e4b194"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:56.401810 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.400168 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-credential-socket" pod "8fef8496-2113-44ff-9694-038139e4b194" (UID: "8fef8496-2113-44ff-9694-038139e4b194"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:56.403698 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.403675 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-istio-data" pod "8fef8496-2113-44ff-9694-038139e4b194" (UID: "8fef8496-2113-44ff-9694-038139e4b194"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:56.404214 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.404193 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fef8496-2113-44ff-9694-038139e4b194-istiod-ca-cert" pod "8fef8496-2113-44ff-9694-038139e4b194" (UID: "8fef8496-2113-44ff-9694-038139e4b194"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:56.407360 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.407331 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-istio-envoy" pod "8fef8496-2113-44ff-9694-038139e4b194" (UID: "8fef8496-2113-44ff-9694-038139e4b194"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:56.407977 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.407941 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fef8496-2113-44ff-9694-038139e4b194-istio-podinfo" pod "8fef8496-2113-44ff-9694-038139e4b194" (UID: "8fef8496-2113-44ff-9694-038139e4b194"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:00:56.409373 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.409338 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fef8496-2113-44ff-9694-038139e4b194-istio-token" pod "8fef8496-2113-44ff-9694-038139e4b194" (UID: "8fef8496-2113-44ff-9694-038139e4b194"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:56.410361 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.410329 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fef8496-2113-44ff-9694-038139e4b194-kube-api-access-rtb7h" pod "8fef8496-2113-44ff-9694-038139e4b194" (UID: "8fef8496-2113-44ff-9694-038139e4b194"). InnerVolumeSpecName "kube-api-access-rtb7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:56.419806 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.419080 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" podStartSLOduration=1.419059025 podStartE2EDuration="1.419059025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:56.416272559 +0000 UTC m=+1361.596169067" watchObservedRunningTime="2026-10-30 21:00:56.419059025 +0000 UTC m=+1361.598955606" Oct 30 21:00:56.452378 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.452311 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" podStartSLOduration=1.45229667 podStartE2EDuration="1.45229667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:56.449319515 +0000 UTC m=+1361.629216035" watchObservedRunningTime="2026-10-30 21:00:56.45229667 +0000 UTC m=+1361.632193177" Oct 30 21:00:56.504867 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.499974 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtb7h\" (UniqueName: \"kubernetes.io/projected/8fef8496-2113-44ff-9694-038139e4b194-kube-api-access-rtb7h\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:56.504867 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.500013 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fef8496-2113-44ff-9694-038139e4b194-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:56.504867 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.500027 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fef8496-2113-44ff-9694-038139e4b194-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:56.504867 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.500040 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:56.504867 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.500053 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:56.504867 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.500067 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:56.504867 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.500079 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fef8496-2113-44ff-9694-038139e4b194-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:56.504867 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.500093 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:56.504867 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:56.500107 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fef8496-2113-44ff-9694-038139e4b194-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:00:57.362942 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:57.362190 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 21:00:57.362942 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:57.362263 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" podUID="cf35d310-f121-41d1-844c-e2bef5d31d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 21:00:57.365054 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:57.365025 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm" Oct 30 21:00:57.386132 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:57.386100 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 21:00:57.386310 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:57.386149 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" podUID="91bae7a3-eafb-4ec3-976b-d885576ecca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 21:00:57.452798 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:57.451634 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm"] Oct 30 21:00:57.462906 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:57.458980 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-955eb7-openshift-default-9b68488fc-66gxm"] Oct 30 21:00:58.356790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:58.356258 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 21:00:58.356790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:58.356332 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" podUID="cf35d310-f121-41d1-844c-e2bef5d31d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 21:00:58.385166 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:58.385129 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 21:00:58.385566 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:58.385193 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" podUID="91bae7a3-eafb-4ec3-976b-d885576ecca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 21:00:58.958793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:58.957802 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fef8496-2113-44ff-9694-038139e4b194" path="/var/lib/kubelet/pods/8fef8496-2113-44ff-9694-038139e4b194/volumes" Oct 30 21:00:59.359845 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:59.359248 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 21:00:59.359845 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:59.359309 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" podUID="cf35d310-f121-41d1-844c-e2bef5d31d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 21:00:59.385394 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:59.385351 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 21:00:59.385941 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:00:59.385405 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" podUID="91bae7a3-eafb-4ec3-976b-d885576ecca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 21:01:00.365792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:00.361177 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 21:01:00.365792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:00.361260 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" podUID="cf35d310-f121-41d1-844c-e2bef5d31d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 21:01:00.420300 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:00.420252 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 21:01:00.420910 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:00.420865 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" podUID="91bae7a3-eafb-4ec3-976b-d885576ecca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 21:01:01.362382 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:01.362346 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 21:01:01.362938 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:01.362903 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" podUID="cf35d310-f121-41d1-844c-e2bef5d31d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 21:01:01.387213 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:01.387176 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 21:01:01.387484 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:01.387457 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" podUID="91bae7a3-eafb-4ec3-976b-d885576ecca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 21:01:01.552924 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:01.552871 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:01.581582 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:01.581477 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:01.631662 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:01.631567 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:02.020908 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:02.020858 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:02.262943 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:02.262896 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:02.323848 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:02.322896 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" podUID="14ba0389-1004-4f23-8560-cf6fe6f597dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.173:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:02.358965 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:02.358928 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 21:01:02.359230 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:02.359203 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" podUID="cf35d310-f121-41d1-844c-e2bef5d31d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 21:01:02.408790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:02.407736 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 21:01:02.408790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:02.407819 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" podUID="91bae7a3-eafb-4ec3-976b-d885576ecca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 21:01:02.514793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:02.514501 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Oct 30 21:01:02.667887 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:02.591966 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 21:01:02.667887 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:02.592588 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 21:01:02.667887 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:02.594152 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:01:02.667887 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:02.594607 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 21:01:02.667887 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:02.643432 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 21:01:02.680841 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:02.668839 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 21:01:02.959068 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:02.918464 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:03.060794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:03.024104 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:03.228853 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:03.200791 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 21:01:03.325896 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:03.325850 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" podUID="14ba0389-1004-4f23-8560-cf6fe6f597dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.173:8000/ready\": context deadline exceeded" Oct 30 21:01:03.340002 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:03.332680 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:01:03.433798 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:03.359212 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 21:01:03.433798 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:03.359274 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" podUID="cf35d310-f121-41d1-844c-e2bef5d31d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 21:01:03.483129 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:03.477490 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 21:01:03.483129 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:03.477585 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" podUID="91bae7a3-eafb-4ec3-976b-d885576ecca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 21:01:03.621793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:03.617860 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:03.654833 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:03.636104 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:03.714752 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:03.700425 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:03.929845 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:03.923067 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 21:01:03.929845 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:03.923728 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 21:01:04.092303 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:04.092257 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:04.102961 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:04.100283 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 21:01:04.102961 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:04.101746 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 21:01:04.181845 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:04.158460 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 21:01:04.259388 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:04.259144 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:01:04.259813 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:04.259792 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 21:01:04.259970 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:04.259856 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 21:01:04.357089 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:04.355817 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 21:01:04.357089 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:04.355869 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" podUID="cf35d310-f121-41d1-844c-e2bef5d31d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 21:01:04.385672 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:04.385611 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 21:01:04.385986 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:04.385947 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" podUID="91bae7a3-eafb-4ec3-976b-d885576ecca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 21:01:05.384934 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:05.384895 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 21:01:05.385480 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:05.384965 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" podUID="91bae7a3-eafb-4ec3-976b-d885576ecca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 21:01:06.357448 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:06.357405 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:01:06.357679 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:06.357471 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" podUID="cf35d310-f121-41d1-844c-e2bef5d31d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:07.383278 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:07.383235 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:01:07.383740 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:07.383301 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" podUID="cf35d310-f121-41d1-844c-e2bef5d31d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:07.384313 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:07.384288 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:01:07.384556 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:07.384536 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:01:07.384643 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:07.384577 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" podUID="91bae7a3-eafb-4ec3-976b-d885576ecca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:07.384697 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:07.384685 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:01:07.394682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:07.394657 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:01:07.629435 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:07.629404 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:01:07.631286 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:07.631265 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:01:07.633926 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:07.633864 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:01:10.054234 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.054185 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm"] Oct 30 21:01:10.054732 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.054428 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm" podUID="e1246b25-a2e4-4021-b38e-64a255ba4a39" containerName="payload-processing" containerID="cri-o://c26ca51e9ae4c1b03b5074fdb226bbb4cf19430eb7e14c75c79c57094da8c7eb" gracePeriod=30 Oct 30 21:01:10.059508 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.059477 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w"] Oct 30 21:01:10.059814 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.059784 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w" podUID="be1606dd-387e-4ab0-aa7f-7e12be2c3323" containerName="payload-pre-processing" containerID="cri-o://95931456f8261191a87a54d27e0aab3225c998165b5e33130796b3a65857e686" gracePeriod=30 Oct 30 21:01:10.437252 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.437216 4848 generic.go:356] "Generic (PLEG): container finished" podID="be1606dd-387e-4ab0-aa7f-7e12be2c3323" containerID="95931456f8261191a87a54d27e0aab3225c998165b5e33130796b3a65857e686" exitCode=0 Oct 30 21:01:10.438143 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.437589 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w" event={"ID":"be1606dd-387e-4ab0-aa7f-7e12be2c3323","Type":"ContainerDied","Data":"95931456f8261191a87a54d27e0aab3225c998165b5e33130796b3a65857e686"} Oct 30 21:01:10.440962 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.440938 4848 generic.go:356] "Generic (PLEG): container finished" podID="e1246b25-a2e4-4021-b38e-64a255ba4a39" containerID="c26ca51e9ae4c1b03b5074fdb226bbb4cf19430eb7e14c75c79c57094da8c7eb" exitCode=0 Oct 30 21:01:10.441144 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.441128 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm" event={"ID":"e1246b25-a2e4-4021-b38e-64a255ba4a39","Type":"ContainerDied","Data":"c26ca51e9ae4c1b03b5074fdb226bbb4cf19430eb7e14c75c79c57094da8c7eb"} Oct 30 21:01:10.606529 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.606498 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm" Oct 30 21:01:10.643678 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.643652 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w" Oct 30 21:01:10.688303 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.688219 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be1606dd-387e-4ab0-aa7f-7e12be2c3323-kube-api-access-xv6b6\" (UniqueName: \"kubernetes.io/projected/be1606dd-387e-4ab0-aa7f-7e12be2c3323-kube-api-access-xv6b6\") pod \"be1606dd-387e-4ab0-aa7f-7e12be2c3323\" (UID: \"be1606dd-387e-4ab0-aa7f-7e12be2c3323\") " Oct 30 21:01:10.688608 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.688584 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1246b25-a2e4-4021-b38e-64a255ba4a39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1246b25-a2e4-4021-b38e-64a255ba4a39-plugins-config-volume\") pod \"e1246b25-a2e4-4021-b38e-64a255ba4a39\" (UID: \"e1246b25-a2e4-4021-b38e-64a255ba4a39\") " Oct 30 21:01:10.688926 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.688900 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1246b25-a2e4-4021-b38e-64a255ba4a39-kube-api-access-kr52n\" (UniqueName: \"kubernetes.io/projected/e1246b25-a2e4-4021-b38e-64a255ba4a39-kube-api-access-kr52n\") pod \"e1246b25-a2e4-4021-b38e-64a255ba4a39\" (UID: \"e1246b25-a2e4-4021-b38e-64a255ba4a39\") " Oct 30 21:01:10.689192 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.689168 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be1606dd-387e-4ab0-aa7f-7e12be2c3323-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be1606dd-387e-4ab0-aa7f-7e12be2c3323-plugins-config-volume\") pod \"be1606dd-387e-4ab0-aa7f-7e12be2c3323\" (UID: \"be1606dd-387e-4ab0-aa7f-7e12be2c3323\") " Oct 30 21:01:10.690529 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.690507 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1246b25-a2e4-4021-b38e-64a255ba4a39-plugins-config-volume" pod "e1246b25-a2e4-4021-b38e-64a255ba4a39" (UID: "e1246b25-a2e4-4021-b38e-64a255ba4a39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:01:10.690857 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.690810 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be1606dd-387e-4ab0-aa7f-7e12be2c3323-plugins-config-volume" pod "be1606dd-387e-4ab0-aa7f-7e12be2c3323" (UID: "be1606dd-387e-4ab0-aa7f-7e12be2c3323"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:01:10.761921 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.757011 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be1606dd-387e-4ab0-aa7f-7e12be2c3323-kube-api-access-xv6b6" pod "be1606dd-387e-4ab0-aa7f-7e12be2c3323" (UID: "be1606dd-387e-4ab0-aa7f-7e12be2c3323"). InnerVolumeSpecName "kube-api-access-xv6b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:10.765980 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.763760 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1246b25-a2e4-4021-b38e-64a255ba4a39-kube-api-access-kr52n" pod "e1246b25-a2e4-4021-b38e-64a255ba4a39" (UID: "e1246b25-a2e4-4021-b38e-64a255ba4a39"). InnerVolumeSpecName "kube-api-access-kr52n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:10.791564 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.791530 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1246b25-a2e4-4021-b38e-64a255ba4a39-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:10.791781 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.791750 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr52n\" (UniqueName: \"kubernetes.io/projected/e1246b25-a2e4-4021-b38e-64a255ba4a39-kube-api-access-kr52n\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:10.791916 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.791901 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be1606dd-387e-4ab0-aa7f-7e12be2c3323-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:10.792232 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:10.792214 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv6b6\" (UniqueName: \"kubernetes.io/projected/be1606dd-387e-4ab0-aa7f-7e12be2c3323-kube-api-access-xv6b6\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:11.449025 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:11.448991 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w" Oct 30 21:01:11.449477 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:11.449023 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w" event={"ID":"be1606dd-387e-4ab0-aa7f-7e12be2c3323","Type":"ContainerDied","Data":"d3c4a8d57e5e594f5dc3f1f7f82e1ae345d4a097a7f3196045c7ec00ea16352b"} Oct 30 21:01:11.449477 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:11.449100 4848 scope.go:122] "RemoveContainer" containerID="95931456f8261191a87a54d27e0aab3225c998165b5e33130796b3a65857e686" Oct 30 21:01:11.450826 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:11.450805 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm" event={"ID":"e1246b25-a2e4-4021-b38e-64a255ba4a39","Type":"ContainerDied","Data":"195da83271242e95b11ba50e6a318b344a36dd82b1767edfd702c3020f2d1afb"} Oct 30 21:01:11.450917 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:11.450851 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm" Oct 30 21:01:11.460184 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:11.460159 4848 scope.go:122] "RemoveContainer" containerID="c26ca51e9ae4c1b03b5074fdb226bbb4cf19430eb7e14c75c79c57094da8c7eb" Oct 30 21:01:11.473229 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:11.473195 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w"] Oct 30 21:01:11.479703 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:11.479672 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b80638-7457cd58d7-fgs2w"] Oct 30 21:01:11.497711 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:11.497668 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm"] Oct 30 21:01:11.501886 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:11.501855 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b80638-567fddf986-fgmdm"] Oct 30 21:01:12.932990 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:12.932957 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be1606dd-387e-4ab0-aa7f-7e12be2c3323" path="/var/lib/kubelet/pods/be1606dd-387e-4ab0-aa7f-7e12be2c3323/volumes" Oct 30 21:01:12.933544 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:12.933524 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1246b25-a2e4-4021-b38e-64a255ba4a39" path="/var/lib/kubelet/pods/e1246b25-a2e4-4021-b38e-64a255ba4a39/volumes" Oct 30 21:01:18.524929 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:18.524891 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m"] Oct 30 21:01:18.526038 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:18.525981 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" podUID="e9b4f116-4af8-4321-887c-1df07ed0cd5d" containerName="istio-proxy" containerID="cri-o://927cacb73efd93ec62c1d8c048842facfa010a9b8b1761539817a79bc2459e60" gracePeriod=30 Oct 30 21:01:19.248231 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:19.247540 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" podUID="14ba0389-1004-4f23-8560-cf6fe6f597dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.173:8000/ready\": context deadline exceeded" Oct 30 21:01:19.267670 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:19.267620 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:19.639104 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:19.639010 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:20.107022 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:20.106970 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:20.162998 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:20.162941 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:20.181581 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:20.181521 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:01:20.181761 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:20.181594 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" podUID="e9b4f116-4af8-4321-887c-1df07ed0cd5d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:01:20.327265 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:20.320818 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g"] Oct 30 21:01:20.327265 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:20.321785 4848 cpu_manager.go:414] "containerMap: removing container" podUID="8fef8496-2113-44ff-9694-038139e4b194" containerName="istio-proxy" Oct 30 21:01:20.342865 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:20.342826 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fef8496-2113-44ff-9694-038139e4b194" containerName="istio-proxy" Oct 30 21:01:20.346795 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:20.343152 4848 cpu_manager.go:414] "containerMap: removing container" podUID="e1246b25-a2e4-4021-b38e-64a255ba4a39" containerName="payload-processing" Oct 30 21:01:20.486794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:20.466826 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1246b25-a2e4-4021-b38e-64a255ba4a39" containerName="payload-processing" Oct 30 21:01:20.491939 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:20.491901 4848 cpu_manager.go:414] "containerMap: removing container" podUID="be1606dd-387e-4ab0-aa7f-7e12be2c3323" containerName="payload-pre-processing" Oct 30 21:01:20.492159 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:20.492141 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be1606dd-387e-4ab0-aa7f-7e12be2c3323" containerName="payload-pre-processing" Oct 30 21:01:20.509803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:20.492268 4848 cpu_manager.go:414] "containerMap: removing container" podUID="8fef8496-2113-44ff-9694-038139e4b194" containerName="istio-proxy" Oct 30 21:01:20.510033 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:20.510007 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fef8496-2113-44ff-9694-038139e4b194" containerName="istio-proxy" Oct 30 21:01:20.592794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:20.530297 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fef8496-2113-44ff-9694-038139e4b194" containerName="istio-proxy" Oct 30 21:01:20.879851 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:20.877884 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1246b25-a2e4-4021-b38e-64a255ba4a39" containerName="payload-processing" Oct 30 21:01:20.879851 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:20.877935 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="be1606dd-387e-4ab0-aa7f-7e12be2c3323" containerName="payload-pre-processing" Oct 30 21:01:20.892517 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:20.688808 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:01:20.896796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:20.740340 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:20.957150 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:20.957114 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 21:01:20.993791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:20.972221 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.082807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.077592 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-podinfo\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.082807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.077678 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-workload-certs\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.082807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.077736 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5bc9955-5961-48d4-aca4-e225572d0efa-istiod-ca-cert\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.082807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.077829 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-workload-socket\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.082807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.077884 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-envoy\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.082807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.077921 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-credential-socket\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.082807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.077976 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-data\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.082807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.078065 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kzp2\" (UniqueName: \"kubernetes.io/projected/f5bc9955-5961-48d4-aca4-e225572d0efa-kube-api-access-5kzp2\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.082807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.078107 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-token\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.088550 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.087177 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-117ee1-openshift-default-dockercfg-lsssc\"" Oct 30 21:01:21.112806 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.104266 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:21.113071 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.113026 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:21.140093 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.130876 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g"] Oct 30 21:01:21.140093 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.130989 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 21:01:21.189807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.175990 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:21.189807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.176523 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 21:01:21.189807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.178953 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-workload-certs\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.189807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.179011 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5bc9955-5961-48d4-aca4-e225572d0efa-istiod-ca-cert\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.189807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.179070 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-workload-socket\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.189807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.179121 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-envoy\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.189807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.179154 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-credential-socket\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.189807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.179200 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-data\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.189807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.179249 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kzp2\" (UniqueName: \"kubernetes.io/projected/f5bc9955-5961-48d4-aca4-e225572d0efa-kube-api-access-5kzp2\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.189807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.179288 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-token\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.189807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.179450 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-podinfo\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.252400 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.239273 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-workload-certs\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.252636 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.241921 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-workload-socket\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.252736 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.242798 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" podUID="14ba0389-1004-4f23-8560-cf6fe6f597dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.173:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:21.268904 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.243401 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5bc9955-5961-48d4-aca4-e225572d0efa-istiod-ca-cert\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.268904 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.243757 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-data\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.268904 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.245072 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-credential-socket\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.291489 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.291432 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Oct 30 21:01:21.564843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.550394 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-podinfo\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.565870 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.565836 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kzp2\" (UniqueName: \"kubernetes.io/projected/f5bc9955-5961-48d4-aca4-e225572d0efa-kube-api-access-5kzp2\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.579485 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.579413 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-envoy\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.756434 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.756357 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-token\") pod \"e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:21.927576 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.927527 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:21.928957 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.928927 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:01:21.929671 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.929648 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 21:01:21.946791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:21.943307 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:22.206171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:22.196356 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:22.206171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:22.196991 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 21:01:22.206171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:22.198587 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:22.250806 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:22.244225 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:22.284969 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:22.283380 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" podUID="14ba0389-1004-4f23-8560-cf6fe6f597dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.173:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:22.284969 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:22.284165 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:01:22.286213 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:22.286142 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": context deadline exceeded" Oct 30 21:01:22.459066 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:22.458963 4848 patch_prober.go:29] interesting pod/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:01:22.459840 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:22.459312 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" podUID="ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:22.538526 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:22.509695 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:22.958165 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:22.953060 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:22.958165 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:22.953450 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 21:01:23.166178 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:01:23.166126 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:01:23.234757 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:23.234653 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:23.271336 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:23.258983 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:23.293215 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:23.271829 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 21:01:23.369822 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:23.285026 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" podUID="14ba0389-1004-4f23-8560-cf6fe6f597dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.173:8000/ready\": context deadline exceeded" Oct 30 21:01:23.456792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:23.452943 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:01:23.456792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:23.453011 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" podUID="cf35d310-f121-41d1-844c-e2bef5d31d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:23.609409 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:23.587997 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 21:01:23.807333 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:23.807298 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:01:23.807624 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:23.807589 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" podUID="91bae7a3-eafb-4ec3-976b-d885576ecca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:23.972803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:23.969998 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:23.999789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:23.984911 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 21:01:24.077525 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.069800 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 21:01:24.161885 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.154527 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 21:01:24.279368 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.266849 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:01:24.279368 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.271080 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 21:01:24.350801 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.343943 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 21:01:24.449235 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.449185 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:01:24.451722 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.451683 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:01:24.452135 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.452109 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:01:24.467173 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.463363 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g"] Oct 30 21:01:24.473856 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.473828 4848 generic.go:356] "Generic (PLEG): container finished" podID="e9b4f116-4af8-4321-887c-1df07ed0cd5d" containerID="927cacb73efd93ec62c1d8c048842facfa010a9b8b1761539817a79bc2459e60" exitCode=0 Oct 30 21:01:24.474875 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.474288 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" event={"ID":"e9b4f116-4af8-4321-887c-1df07ed0cd5d","Type":"ContainerDied","Data":"927cacb73efd93ec62c1d8c048842facfa010a9b8b1761539817a79bc2459e60"} Oct 30 21:01:24.482297 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.482265 4848 scope.go:122] "RemoveContainer" containerID="cfa3bcfdab31604d7d29849a2da76015971a05de5efb266754a476a3b0e5fab7" Oct 30 21:01:24.635830 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.635791 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 21:01:24.823548 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.823452 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istiod-ca-cert\") pod \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " Oct 30 21:01:24.823548 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.823499 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9b4f116-4af8-4321-887c-1df07ed0cd5d-kube-api-access-vcf5t\" (UniqueName: \"kubernetes.io/projected/e9b4f116-4af8-4321-887c-1df07ed0cd5d-kube-api-access-vcf5t\") pod \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " Oct 30 21:01:24.823548 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.823528 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-podinfo\") pod \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " Oct 30 21:01:24.823881 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.823577 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-workload-socket\") pod \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " Oct 30 21:01:24.823881 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.823626 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-token\" (UniqueName: \"kubernetes.io/projected/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-token\") pod \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " Oct 30 21:01:24.823881 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.823676 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-credential-socket\") pod \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " Oct 30 21:01:24.823881 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.823733 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-workload-certs\") pod \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " Oct 30 21:01:24.823881 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.823834 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-data\") pod \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " Oct 30 21:01:24.824165 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.823908 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-workload-socket" pod "e9b4f116-4af8-4321-887c-1df07ed0cd5d" (UID: "e9b4f116-4af8-4321-887c-1df07ed0cd5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:24.824165 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.823920 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-envoy\") pod \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\" (UID: \"e9b4f116-4af8-4321-887c-1df07ed0cd5d\") " Oct 30 21:01:24.824165 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.823974 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istiod-ca-cert" pod "e9b4f116-4af8-4321-887c-1df07ed0cd5d" (UID: "e9b4f116-4af8-4321-887c-1df07ed0cd5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:01:24.824341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.824208 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-credential-socket" pod "e9b4f116-4af8-4321-887c-1df07ed0cd5d" (UID: "e9b4f116-4af8-4321-887c-1df07ed0cd5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:24.824634 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.824610 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-workload-certs" pod "e9b4f116-4af8-4321-887c-1df07ed0cd5d" (UID: "e9b4f116-4af8-4321-887c-1df07ed0cd5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:24.824859 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.824722 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-data" pod "e9b4f116-4af8-4321-887c-1df07ed0cd5d" (UID: "e9b4f116-4af8-4321-887c-1df07ed0cd5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:24.825019 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.824688 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:24.825368 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.825333 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:24.825605 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.825588 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:24.826445 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.826409 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-envoy" pod "e9b4f116-4af8-4321-887c-1df07ed0cd5d" (UID: "e9b4f116-4af8-4321-887c-1df07ed0cd5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:24.826612 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.826586 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b4f116-4af8-4321-887c-1df07ed0cd5d-kube-api-access-vcf5t" pod "e9b4f116-4af8-4321-887c-1df07ed0cd5d" (UID: "e9b4f116-4af8-4321-887c-1df07ed0cd5d"). InnerVolumeSpecName "kube-api-access-vcf5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:24.827207 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.827179 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-token" pod "e9b4f116-4af8-4321-887c-1df07ed0cd5d" (UID: "e9b4f116-4af8-4321-887c-1df07ed0cd5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:24.827405 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.827385 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-podinfo" pod "e9b4f116-4af8-4321-887c-1df07ed0cd5d" (UID: "e9b4f116-4af8-4321-887c-1df07ed0cd5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:01:24.926949 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.926914 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcf5t\" (UniqueName: \"kubernetes.io/projected/e9b4f116-4af8-4321-887c-1df07ed0cd5d-kube-api-access-vcf5t\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:24.926949 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.926950 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:24.927274 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.926965 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:24.927274 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.926979 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:24.927274 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.926993 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:24.927274 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:24.927007 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9b4f116-4af8-4321-887c-1df07ed0cd5d-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:01:25.485317 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:25.485288 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" Oct 30 21:01:25.485317 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:25.485305 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m" event={"ID":"e9b4f116-4af8-4321-887c-1df07ed0cd5d","Type":"ContainerDied","Data":"64e017c899de594f9630513f5a7cba7c37e09c4a76343108cdb8a7093480da7e"} Oct 30 21:01:25.485897 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:25.485366 4848 scope.go:122] "RemoveContainer" containerID="927cacb73efd93ec62c1d8c048842facfa010a9b8b1761539817a79bc2459e60" Oct 30 21:01:25.487277 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:25.487246 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" event={"ID":"f5bc9955-5961-48d4-aca4-e225572d0efa","Type":"ContainerStarted","Data":"24a742a7fc2d50572a9fca0826f29b959e56c3d96a6fbfa3b4fc752392c3af41"} Oct 30 21:01:25.487400 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:25.487312 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" event={"ID":"f5bc9955-5961-48d4-aca4-e225572d0efa","Type":"ContainerStarted","Data":"af0a0be4a7680894ff75caebafd46279d019edfb3b14c43368e4f9c6f1500581"} Oct 30 21:01:25.513474 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:25.513422 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" podStartSLOduration=5.513408792 podStartE2EDuration="5.513408792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:25.51058022 +0000 UTC m=+1390.690476728" watchObservedRunningTime="2026-10-30 21:01:25.513408792 +0000 UTC m=+1390.693305299" Oct 30 21:01:25.528399 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:25.528368 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m"] Oct 30 21:01:25.535077 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:25.535048 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b80638-openshift-default-66884b6f5f-2k85m"] Oct 30 21:01:25.966566 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:25.966529 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:25.972490 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:25.972466 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:26.495455 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:26.495425 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:26.496454 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:26.496438 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:01:26.934879 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:26.934842 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9b4f116-4af8-4321-887c-1df07ed0cd5d" path="/var/lib/kubelet/pods/e9b4f116-4af8-4321-887c-1df07ed0cd5d/volumes" Oct 30 21:01:29.184906 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.184874 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6"] Oct 30 21:01:29.185525 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.185510 4848 cpu_manager.go:414] "containerMap: removing container" podUID="e9b4f116-4af8-4321-887c-1df07ed0cd5d" containerName="istio-proxy" Oct 30 21:01:29.185582 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.185526 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9b4f116-4af8-4321-887c-1df07ed0cd5d" containerName="istio-proxy" Oct 30 21:01:29.185582 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.185543 4848 cpu_manager.go:414] "containerMap: removing container" podUID="e9b4f116-4af8-4321-887c-1df07ed0cd5d" containerName="istio-proxy" Oct 30 21:01:29.185582 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.185550 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9b4f116-4af8-4321-887c-1df07ed0cd5d" containerName="istio-proxy" Oct 30 21:01:29.185681 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.185642 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9b4f116-4af8-4321-887c-1df07ed0cd5d" containerName="istio-proxy" Oct 30 21:01:29.185681 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.185652 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9b4f116-4af8-4321-887c-1df07ed0cd5d" containerName="istio-proxy" Oct 30 21:01:29.185681 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.185663 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fef8496-2113-44ff-9694-038139e4b194" containerName="istio-proxy" Oct 30 21:01:29.190881 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.190861 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6" Oct 30 21:01:29.199019 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.198993 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-117ee1\"" Oct 30 21:01:29.200506 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.200479 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6"] Oct 30 21:01:29.208694 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.208669 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv"] Oct 30 21:01:29.214268 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.214240 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv" Oct 30 21:01:29.229786 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.224494 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-117ee1-dockercfg-5cdpv\"" Oct 30 21:01:29.236034 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.236003 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv"] Oct 30 21:01:29.302264 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.302230 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95hkv\" (UniqueName: \"kubernetes.io/projected/89b53a4d-165c-4b95-b0d2-913b50203b36-kube-api-access-95hkv\") pod \"payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv\" (UID: \"89b53a4d-165c-4b95-b0d2-913b50203b36\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv" Oct 30 21:01:29.302469 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.302452 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9jp6\" (UniqueName: \"kubernetes.io/projected/ea1a06fd-c761-43cb-b31c-47a22546106d-kube-api-access-m9jp6\") pod \"payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6\" (UID: \"ea1a06fd-c761-43cb-b31c-47a22546106d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6" Oct 30 21:01:29.302513 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.302480 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea1a06fd-c761-43cb-b31c-47a22546106d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6\" (UID: \"ea1a06fd-c761-43cb-b31c-47a22546106d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6" Oct 30 21:01:29.302555 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.302509 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89b53a4d-165c-4b95-b0d2-913b50203b36-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv\" (UID: \"89b53a4d-165c-4b95-b0d2-913b50203b36\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv" Oct 30 21:01:29.408790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.403669 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9jp6\" (UniqueName: \"kubernetes.io/projected/ea1a06fd-c761-43cb-b31c-47a22546106d-kube-api-access-m9jp6\") pod \"payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6\" (UID: \"ea1a06fd-c761-43cb-b31c-47a22546106d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6" Oct 30 21:01:29.408790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.403729 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea1a06fd-c761-43cb-b31c-47a22546106d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6\" (UID: \"ea1a06fd-c761-43cb-b31c-47a22546106d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6" Oct 30 21:01:29.408790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.403795 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89b53a4d-165c-4b95-b0d2-913b50203b36-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv\" (UID: \"89b53a4d-165c-4b95-b0d2-913b50203b36\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv" Oct 30 21:01:29.408790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.403885 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95hkv\" (UniqueName: \"kubernetes.io/projected/89b53a4d-165c-4b95-b0d2-913b50203b36-kube-api-access-95hkv\") pod \"payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv\" (UID: \"89b53a4d-165c-4b95-b0d2-913b50203b36\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv" Oct 30 21:01:29.408790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.405461 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea1a06fd-c761-43cb-b31c-47a22546106d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6\" (UID: \"ea1a06fd-c761-43cb-b31c-47a22546106d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6" Oct 30 21:01:29.408790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.406213 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89b53a4d-165c-4b95-b0d2-913b50203b36-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv\" (UID: \"89b53a4d-165c-4b95-b0d2-913b50203b36\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv" Oct 30 21:01:29.441832 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.430958 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9jp6\" (UniqueName: \"kubernetes.io/projected/ea1a06fd-c761-43cb-b31c-47a22546106d-kube-api-access-m9jp6\") pod \"payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6\" (UID: \"ea1a06fd-c761-43cb-b31c-47a22546106d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6" Oct 30 21:01:29.441832 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.437993 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95hkv\" (UniqueName: \"kubernetes.io/projected/89b53a4d-165c-4b95-b0d2-913b50203b36-kube-api-access-95hkv\") pod \"payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv\" (UID: \"89b53a4d-165c-4b95-b0d2-913b50203b36\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv" Oct 30 21:01:29.535789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.523046 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6" Oct 30 21:01:29.564344 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:29.562171 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv" Oct 30 21:01:30.022682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:30.022643 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv"] Oct 30 21:01:30.027557 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:30.027518 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6"] Oct 30 21:01:30.555747 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:30.555710 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6" event={"ID":"ea1a06fd-c761-43cb-b31c-47a22546106d","Type":"ContainerStarted","Data":"ac58b98a77aaca23aa4a1d0faa6fa734634154d1be432fc13a8b9c5b787f4340"} Oct 30 21:01:30.556257 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:30.555850 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6" event={"ID":"ea1a06fd-c761-43cb-b31c-47a22546106d","Type":"ContainerStarted","Data":"a26f6faba6ff3e75dcaa39189fb6674d609a0d2d5fff77b54f6df24f86d43e2f"} Oct 30 21:01:30.556257 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:30.555914 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6" Oct 30 21:01:30.557605 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:30.557581 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv" event={"ID":"89b53a4d-165c-4b95-b0d2-913b50203b36","Type":"ContainerStarted","Data":"3ca477fd14641c45b4dbfef573adbfb96a3d44c7635a7b3bcd50d58ad8a8eb14"} Oct 30 21:01:30.557723 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:30.557624 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv" event={"ID":"89b53a4d-165c-4b95-b0d2-913b50203b36","Type":"ContainerStarted","Data":"7bcd85333b569d83fcdb84afb60823d5601f491facc5723c5835a4518f4d6602"} Oct 30 21:01:30.557723 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:30.557714 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv" Oct 30 21:01:30.576477 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:30.576433 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6" podStartSLOduration=1.5764213169999999 podStartE2EDuration="1.576421317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:30.572279628 +0000 UTC m=+1395.752176136" watchObservedRunningTime="2026-10-30 21:01:30.576421317 +0000 UTC m=+1395.756317823" Oct 30 21:01:30.592576 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:30.592531 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv" podStartSLOduration=1.592520002 podStartE2EDuration="1.592520002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:30.58992994 +0000 UTC m=+1395.769826449" watchObservedRunningTime="2026-10-30 21:01:30.592520002 +0000 UTC m=+1395.772416509" Oct 30 21:01:35.488395 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:35.488353 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq"] Oct 30 21:01:35.496498 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:35.495895 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" Oct 30 21:01:35.503546 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:35.503515 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq"] Oct 30 21:01:35.503743 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:35.503727 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-117ee1-serving-cert\"" Oct 30 21:01:35.565393 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:35.565350 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq\" (UID: \"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" Oct 30 21:01:35.565560 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:35.565402 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-metrics-certs\") pod \"maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq\" (UID: \"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" Oct 30 21:01:35.565560 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:35.565532 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt2n9\" (UniqueName: \"kubernetes.io/projected/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-kube-api-access-jt2n9\") pod \"maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq\" (UID: \"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" Oct 30 21:01:35.565659 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:35.565582 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq\" (UID: \"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" Oct 30 21:01:35.685398 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:35.666813 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq\" (UID: \"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" Oct 30 21:01:35.689725 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:35.686747 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-metrics-certs\") pod \"maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq\" (UID: \"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" Oct 30 21:01:35.690221 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:35.690120 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt2n9\" (UniqueName: \"kubernetes.io/projected/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-kube-api-access-jt2n9\") pod \"maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq\" (UID: \"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" Oct 30 21:01:35.690535 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:35.690488 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq\" (UID: \"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" Oct 30 21:01:35.691952 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:35.691640 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq\" (UID: \"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" Oct 30 21:01:35.692222 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:35.692200 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq\" (UID: \"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" Oct 30 21:01:35.695469 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:35.695444 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-metrics-certs\") pod \"maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq\" (UID: \"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" Oct 30 21:01:35.708504 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:35.708437 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt2n9\" (UniqueName: \"kubernetes.io/projected/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-kube-api-access-jt2n9\") pod \"maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq\" (UID: \"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" Oct 30 21:01:35.822814 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:35.822720 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" Oct 30 21:01:36.111021 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:36.110982 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq"] Oct 30 21:01:36.680136 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:36.680007 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" event={"ID":"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e","Type":"ContainerStarted","Data":"39a08fc8479a8e6fa51c25147b4a9405c321b6aa37d8423b5c123a9665e63d19"} Oct 30 21:01:37.688635 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:37.688595 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" event={"ID":"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e","Type":"ContainerStarted","Data":"46ad8526774a685cd50ed188682d7aabbd09a7f33575e89a257c9c6f8a90cbf8"} Oct 30 21:01:37.689091 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:37.688674 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" Oct 30 21:01:37.708107 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:37.708059 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" podStartSLOduration=2.111381308 podStartE2EDuration="2.708044908s" totalImagesPullingTime="596.6636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:37.704297874 +0000 UTC m=+1402.884194380" watchObservedRunningTime="2026-10-30 21:01:37.708044908 +0000 UTC m=+1402.887941414" Oct 30 21:01:41.600957 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:41.600827 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv" Oct 30 21:01:41.607793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:41.605925 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6" Oct 30 21:01:41.967446 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:41.967061 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:01:42.001097 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:42.001040 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:42.075953 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:42.074879 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:42.159009 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:42.157867 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:42.285847 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:42.267126 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:42.285847 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:42.274873 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:42.434803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:42.430857 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" podUID="14ba0389-1004-4f23-8560-cf6fe6f597dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.173:8000/ready\": context deadline exceeded" Oct 30 21:01:42.484055 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:42.484002 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:43.700722 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:43.700691 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" Oct 30 21:01:52.513688 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:01:52.513648 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:01:54.049266 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.049229 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx"] Oct 30 21:01:54.053609 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.053585 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx" Oct 30 21:01:54.074688 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.074656 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-fa48f3\"" Oct 30 21:01:54.076248 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.076222 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-fa48f3-dockercfg-nw5qh\"" Oct 30 21:01:54.082498 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.082465 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx"] Oct 30 21:01:54.089952 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.089923 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc"] Oct 30 21:01:54.095270 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.095222 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc" Oct 30 21:01:54.107616 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.106890 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc"] Oct 30 21:01:54.137345 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.137311 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95jw4\" (UniqueName: \"kubernetes.io/projected/cc8792ce-9e3b-47c0-93a5-e5a7ded65612-kube-api-access-95jw4\") pod \"payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx\" (UID: \"cc8792ce-9e3b-47c0-93a5-e5a7ded65612\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx" Oct 30 21:01:54.137517 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.137428 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ntv5\" (UniqueName: \"kubernetes.io/projected/5dd5f82e-2c41-4971-ad3b-2bb874ae225c-kube-api-access-9ntv5\") pod \"payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc\" (UID: \"5dd5f82e-2c41-4971-ad3b-2bb874ae225c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc" Oct 30 21:01:54.137517 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.137497 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd5f82e-2c41-4971-ad3b-2bb874ae225c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc\" (UID: \"5dd5f82e-2c41-4971-ad3b-2bb874ae225c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc" Oct 30 21:01:54.137681 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.137544 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8792ce-9e3b-47c0-93a5-e5a7ded65612-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx\" (UID: \"cc8792ce-9e3b-47c0-93a5-e5a7ded65612\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx" Oct 30 21:01:54.238861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.238821 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95jw4\" (UniqueName: \"kubernetes.io/projected/cc8792ce-9e3b-47c0-93a5-e5a7ded65612-kube-api-access-95jw4\") pod \"payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx\" (UID: \"cc8792ce-9e3b-47c0-93a5-e5a7ded65612\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx" Oct 30 21:01:54.239075 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.238901 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ntv5\" (UniqueName: \"kubernetes.io/projected/5dd5f82e-2c41-4971-ad3b-2bb874ae225c-kube-api-access-9ntv5\") pod \"payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc\" (UID: \"5dd5f82e-2c41-4971-ad3b-2bb874ae225c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc" Oct 30 21:01:54.239320 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.239285 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd5f82e-2c41-4971-ad3b-2bb874ae225c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc\" (UID: \"5dd5f82e-2c41-4971-ad3b-2bb874ae225c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc" Oct 30 21:01:54.239415 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.239335 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8792ce-9e3b-47c0-93a5-e5a7ded65612-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx\" (UID: \"cc8792ce-9e3b-47c0-93a5-e5a7ded65612\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx" Oct 30 21:01:54.240324 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.240294 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8792ce-9e3b-47c0-93a5-e5a7ded65612-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx\" (UID: \"cc8792ce-9e3b-47c0-93a5-e5a7ded65612\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx" Oct 30 21:01:54.240439 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.240352 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd5f82e-2c41-4971-ad3b-2bb874ae225c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc\" (UID: \"5dd5f82e-2c41-4971-ad3b-2bb874ae225c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc" Oct 30 21:01:54.253470 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.253425 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ntv5\" (UniqueName: \"kubernetes.io/projected/5dd5f82e-2c41-4971-ad3b-2bb874ae225c-kube-api-access-9ntv5\") pod \"payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc\" (UID: \"5dd5f82e-2c41-4971-ad3b-2bb874ae225c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc" Oct 30 21:01:54.254058 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.254036 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95jw4\" (UniqueName: \"kubernetes.io/projected/cc8792ce-9e3b-47c0-93a5-e5a7ded65612-kube-api-access-95jw4\") pod \"payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx\" (UID: \"cc8792ce-9e3b-47c0-93a5-e5a7ded65612\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx" Oct 30 21:01:54.287734 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.287701 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9"] Oct 30 21:01:54.292095 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.292070 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.298590 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.298562 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-176db5-openshift-default-dockercfg-zm5vp\"" Oct 30 21:01:54.313301 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.313265 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9"] Oct 30 21:01:54.364803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.359126 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-data\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.364803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.359213 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-token\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.364803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.359253 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-workload-socket\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.364803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.359316 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-envoy\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.364803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.359484 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-workload-certs\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.364803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.359565 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istiod-ca-cert\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.364803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.359628 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j9f5\" (UniqueName: \"kubernetes.io/projected/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-kube-api-access-5j9f5\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.364803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.359678 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-podinfo\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.364803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.359733 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-credential-socket\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.392791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.391611 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx" Oct 30 21:01:54.452798 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.451516 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc" Oct 30 21:01:54.521560 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.477270 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-envoy\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.521560 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.477371 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-workload-certs\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.521560 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.477455 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istiod-ca-cert\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.521560 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.477517 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j9f5\" (UniqueName: \"kubernetes.io/projected/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-kube-api-access-5j9f5\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.521560 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.477560 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-podinfo\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.521560 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.477613 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-credential-socket\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.521560 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.477671 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-data\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.521560 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.477737 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-token\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.521560 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.477790 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-workload-socket\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.521560 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.478230 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-workload-socket\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.521560 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.488165 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-data\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.521560 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.488394 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-credential-socket\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.521560 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.488602 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-workload-certs\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.521560 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.489963 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istiod-ca-cert\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.522516 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.522275 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j9f5\" (UniqueName: \"kubernetes.io/projected/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-kube-api-access-5j9f5\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.651516 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.651279 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-token\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.681831 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.681747 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-envoy\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.696464 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.696378 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-podinfo\") pod \"e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:54.910136 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:54.910098 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:55.026683 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:55.026652 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc"] Oct 30 21:01:55.027812 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:01:55.027784 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd5f82e_2c41_4971_ad3b_2bb874ae225c.slice/crio-dea3e64d7ab1b5dd58d43435bc839622b7e39391302ebb4fcc798714b370a6a2 WatchSource:0}: Error finding container dea3e64d7ab1b5dd58d43435bc839622b7e39391302ebb4fcc798714b370a6a2: Status 404 returned error can't find the container with id dea3e64d7ab1b5dd58d43435bc839622b7e39391302ebb4fcc798714b370a6a2 Oct 30 21:01:55.066640 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:55.066612 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx"] Oct 30 21:01:55.240625 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:55.240513 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx" event={"ID":"cc8792ce-9e3b-47c0-93a5-e5a7ded65612","Type":"ContainerStarted","Data":"59a06d7304d3ab305cb5525e4897874ed7220158e91415efdc30d2a2049252bd"} Oct 30 21:01:55.248003 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:55.247957 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc" event={"ID":"5dd5f82e-2c41-4971-ad3b-2bb874ae225c","Type":"ContainerStarted","Data":"dea3e64d7ab1b5dd58d43435bc839622b7e39391302ebb4fcc798714b370a6a2"} Oct 30 21:01:55.309009 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:55.308676 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9"] Oct 30 21:01:55.365857 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:55.365819 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:01:55.366114 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:55.366092 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:01:55.366242 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:55.366227 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:01:56.400851 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:56.396818 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx" event={"ID":"cc8792ce-9e3b-47c0-93a5-e5a7ded65612","Type":"ContainerStarted","Data":"755b84fdf420ec8cbc20670ce5014fe6244976998d919e9a5ca3aeb21c6f660f"} Oct 30 21:01:56.400851 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:56.398027 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx" Oct 30 21:01:56.434839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:56.425276 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc" event={"ID":"5dd5f82e-2c41-4971-ad3b-2bb874ae225c","Type":"ContainerStarted","Data":"1aff8a5fe1e76dfc9e1741f273b325000ade82312c31f1e07f1e121fae3d2fba"} Oct 30 21:01:56.434839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:56.426150 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc" Oct 30 21:01:56.472793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:56.450199 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" event={"ID":"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8","Type":"ContainerStarted","Data":"1d8f8d4ca07607946b6bd623c37e21cceed8ec27b5bc91aea6e27c4d97e51d13"} Oct 30 21:01:56.472793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:56.450270 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" event={"ID":"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8","Type":"ContainerStarted","Data":"997baa0be6e5a9cb9a1000c772ce8ad14a6bb86506f2daa8f9487059f88061c2"} Oct 30 21:01:56.525821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:56.525735 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx" podStartSLOduration=2.525715862 podStartE2EDuration="2.525715862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:56.44661307 +0000 UTC m=+1421.626509579" watchObservedRunningTime="2026-10-30 21:01:56.525715862 +0000 UTC m=+1421.705612387" Oct 30 21:01:56.541300 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:56.528206 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc" podStartSLOduration=2.528192842 podStartE2EDuration="2.528192842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:56.486943783 +0000 UTC m=+1421.666840292" watchObservedRunningTime="2026-10-30 21:01:56.528192842 +0000 UTC m=+1421.708089352" Oct 30 21:01:56.573126 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:56.573057 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" podStartSLOduration=2.573034216 podStartE2EDuration="2.573034216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:01:56.525366057 +0000 UTC m=+1421.705262565" watchObservedRunningTime="2026-10-30 21:01:56.573034216 +0000 UTC m=+1421.752930720" Oct 30 21:01:56.911791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:56.911574 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:56.944099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:56.937137 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:01:56.993242 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:56.991916 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 21:01:57.073681 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:57.073623 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 21:01:57.169274 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:57.169127 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:57.294800 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:57.292912 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 21:01:57.430861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:57.428869 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" podUID="14ba0389-1004-4f23-8560-cf6fe6f597dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.173:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:57.953791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:57.935142 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:57.953791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:57.947904 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:57.953791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:57.948272 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 21:01:58.007791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:58.000850 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:58.007791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:58.001413 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 21:01:58.112827 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:58.101069 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:58.160816 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:58.135100 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 21:01:58.194466 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:58.192929 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:58.194466 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:58.193493 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 21:01:58.342490 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:58.340966 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:58.342490 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:58.341622 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 21:01:58.474038 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:58.448342 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" podUID="14ba0389-1004-4f23-8560-cf6fe6f597dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.173:8000/ready\": context deadline exceeded" Oct 30 21:01:58.636842 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:58.634849 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:01:58.761792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:58.743816 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:58.784791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:58.778014 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:01:58.927794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:58.925092 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 21:01:58.985804 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:58.969976 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:01:59.055434 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:59.055396 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:01:59.056963 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:59.056935 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 21:01:59.062410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:59.062377 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 21:01:59.192582 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:59.192460 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 21:01:59.225500 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:01:59.204017 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 21:02:07.908301 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:07.908268 4848 generic.go:356] "Generic (PLEG): container finished" podID="f5bc9955-5961-48d4-aca4-e225572d0efa" containerID="24a742a7fc2d50572a9fca0826f29b959e56c3d96a6fbfa3b4fc752392c3af41" exitCode=0 Oct 30 21:02:07.908821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:07.908327 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" event={"ID":"f5bc9955-5961-48d4-aca4-e225572d0efa","Type":"ContainerDied","Data":"24a742a7fc2d50572a9fca0826f29b959e56c3d96a6fbfa3b4fc752392c3af41"} Oct 30 21:02:07.908821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:07.908760 4848 scope.go:122] "RemoveContainer" containerID="24a742a7fc2d50572a9fca0826f29b959e56c3d96a6fbfa3b4fc752392c3af41" Oct 30 21:02:07.910324 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:07.910297 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:02:07.910398 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:07.910363 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:02:07.910398 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:07.910395 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:02:08.370779 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:08.370726 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9"] Oct 30 21:02:08.375439 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:08.375410 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" Oct 30 21:02:08.387069 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:08.386636 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-176db5-serving-cert\"" Oct 30 21:02:08.395334 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:08.394982 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9"] Oct 30 21:02:08.478103 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:08.478071 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fca980-3e69-481f-a119-c4e4c2d56d5c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9\" (UID: \"b0fca980-3e69-481f-a119-c4e4c2d56d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" Oct 30 21:02:08.478309 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:08.478284 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbwnn\" (UniqueName: \"kubernetes.io/projected/b0fca980-3e69-481f-a119-c4e4c2d56d5c-kube-api-access-wbwnn\") pod \"maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9\" (UID: \"b0fca980-3e69-481f-a119-c4e4c2d56d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" Oct 30 21:02:08.478378 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:08.478354 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fca980-3e69-481f-a119-c4e4c2d56d5c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9\" (UID: \"b0fca980-3e69-481f-a119-c4e4c2d56d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" Oct 30 21:02:08.478489 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:08.478469 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fca980-3e69-481f-a119-c4e4c2d56d5c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9\" (UID: \"b0fca980-3e69-481f-a119-c4e4c2d56d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" Oct 30 21:02:08.584786 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:08.583666 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fca980-3e69-481f-a119-c4e4c2d56d5c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9\" (UID: \"b0fca980-3e69-481f-a119-c4e4c2d56d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" Oct 30 21:02:08.584786 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:08.583888 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbwnn\" (UniqueName: \"kubernetes.io/projected/b0fca980-3e69-481f-a119-c4e4c2d56d5c-kube-api-access-wbwnn\") pod \"maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9\" (UID: \"b0fca980-3e69-481f-a119-c4e4c2d56d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" Oct 30 21:02:08.584786 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:08.583980 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fca980-3e69-481f-a119-c4e4c2d56d5c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9\" (UID: \"b0fca980-3e69-481f-a119-c4e4c2d56d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" Oct 30 21:02:08.584786 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:08.584046 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fca980-3e69-481f-a119-c4e4c2d56d5c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9\" (UID: \"b0fca980-3e69-481f-a119-c4e4c2d56d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" Oct 30 21:02:08.593511 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:08.593478 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fca980-3e69-481f-a119-c4e4c2d56d5c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9\" (UID: \"b0fca980-3e69-481f-a119-c4e4c2d56d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" Oct 30 21:02:08.597990 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:08.596853 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fca980-3e69-481f-a119-c4e4c2d56d5c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9\" (UID: \"b0fca980-3e69-481f-a119-c4e4c2d56d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" Oct 30 21:02:08.597990 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:08.597920 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fca980-3e69-481f-a119-c4e4c2d56d5c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9\" (UID: \"b0fca980-3e69-481f-a119-c4e4c2d56d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" Oct 30 21:02:08.638882 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:08.638690 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbwnn\" (UniqueName: \"kubernetes.io/projected/b0fca980-3e69-481f-a119-c4e4c2d56d5c-kube-api-access-wbwnn\") pod \"maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9\" (UID: \"b0fca980-3e69-481f-a119-c4e4c2d56d5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" Oct 30 21:02:08.730088 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:08.730051 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" Oct 30 21:02:08.734200 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:08.734177 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc" Oct 30 21:02:08.746314 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:08.746292 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx" Oct 30 21:02:08.947005 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:08.946378 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" event={"ID":"f5bc9955-5961-48d4-aca4-e225572d0efa","Type":"ContainerStarted","Data":"92824b159b6419959779f749ad35eaad08ebfc6015fbbeac7be58f431c6c7e48"} Oct 30 21:02:09.041729 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:09.041692 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 21:02:09.044069 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:09.044029 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" podUID="f5bc9955-5961-48d4-aca4-e225572d0efa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 21:02:09.044518 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:09.044496 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:02:09.131998 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:09.110503 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9"] Oct 30 21:02:09.953810 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:09.953753 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" event={"ID":"b0fca980-3e69-481f-a119-c4e4c2d56d5c","Type":"ContainerStarted","Data":"fe2422904974da9b08a24819ac68b739c5cac545279b83e57560557bfa9ccd26"} Oct 30 21:02:09.954197 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:09.953835 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" event={"ID":"b0fca980-3e69-481f-a119-c4e4c2d56d5c","Type":"ContainerStarted","Data":"be09eea135ec163cab029b9ab016755a362f7f4944d1a2e881ae4dc859c0054f"} Oct 30 21:02:09.954197 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:09.954083 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" Oct 30 21:02:09.967156 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:09.967129 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 21:02:09.967314 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:09.967177 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" podUID="f5bc9955-5961-48d4-aca4-e225572d0efa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 21:02:09.977122 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:09.977072 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" podStartSLOduration=1.420379465 podStartE2EDuration="1.977058296s" totalImagesPullingTime="556.678831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:09.97348508 +0000 UTC m=+1435.153381588" watchObservedRunningTime="2026-10-30 21:02:09.977058296 +0000 UTC m=+1435.156954803" Oct 30 21:02:10.966481 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:10.966444 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 21:02:10.966879 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:10.966501 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" podUID="f5bc9955-5961-48d4-aca4-e225572d0efa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 21:02:11.181289 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.181257 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2"] Oct 30 21:02:11.186951 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.186924 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.226795 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.217821 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-8fa15b-openshift-default-dockercfg-fdxrv\"" Oct 30 21:02:11.239815 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.233217 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2"] Oct 30 21:02:11.239815 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.234097 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2cd625d-00dd-459b-827a-510475c4b444-istio-podinfo\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.239815 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.234155 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-workload-socket\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.239815 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.234181 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-workload-certs\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.239815 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.234236 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2cd625d-00dd-459b-827a-510475c4b444-istio-token\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.239815 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.234284 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-istio-envoy\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.239815 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.234403 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-credential-socket\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.239815 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.234496 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnpxg\" (UniqueName: \"kubernetes.io/projected/a2cd625d-00dd-459b-827a-510475c4b444-kube-api-access-hnpxg\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.239815 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.234525 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-istio-data\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.239815 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.234617 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2cd625d-00dd-459b-827a-510475c4b444-istiod-ca-cert\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.352791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.344938 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-credential-socket\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.352791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.345049 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnpxg\" (UniqueName: \"kubernetes.io/projected/a2cd625d-00dd-459b-827a-510475c4b444-kube-api-access-hnpxg\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.352791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.345083 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-istio-data\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.352791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.345164 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2cd625d-00dd-459b-827a-510475c4b444-istiod-ca-cert\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.352791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.345259 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2cd625d-00dd-459b-827a-510475c4b444-istio-podinfo\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.352791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.345298 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-workload-socket\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.352791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.345323 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-workload-certs\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.352791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.345375 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2cd625d-00dd-459b-827a-510475c4b444-istio-token\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.352791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.345420 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-istio-envoy\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.404792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.371611 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2cd625d-00dd-459b-827a-510475c4b444-istiod-ca-cert\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.404792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.371911 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-credential-socket\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.404792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.372460 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-istio-data\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.404792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.372721 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-workload-socket\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.404792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.373596 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-workload-certs\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.404792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.376736 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2cd625d-00dd-459b-827a-510475c4b444-istio-podinfo\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.404792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.392097 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnpxg\" (UniqueName: \"kubernetes.io/projected/a2cd625d-00dd-459b-827a-510475c4b444-kube-api-access-hnpxg\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.431844 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.427354 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2cd625d-00dd-459b-827a-510475c4b444-istio-token\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.456791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.443117 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-istio-envoy\") pod \"e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.503800 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.495869 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:02:11.552629 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.549147 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:11.966906 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.966863 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 21:02:11.967417 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:11.966934 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" podUID="f5bc9955-5961-48d4-aca4-e225572d0efa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 21:02:12.034661 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:12.034617 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2"] Oct 30 21:02:12.036819 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:02:12.036745 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2cd625d_00dd_459b_827a_510475c4b444.slice/crio-9e25b33fa560c786b44b5633e4058bb8328d488faee1fcaa9ecfdad3faa8db67 WatchSource:0}: Error finding container 9e25b33fa560c786b44b5633e4058bb8328d488faee1fcaa9ecfdad3faa8db67: Status 404 returned error can't find the container with id 9e25b33fa560c786b44b5633e4058bb8328d488faee1fcaa9ecfdad3faa8db67 Oct 30 21:02:12.040058 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:12.040016 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:02:12.040154 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:12.040114 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:02:12.040195 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:12.040156 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:02:12.967022 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:12.966985 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 21:02:12.967436 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:12.967055 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" podUID="f5bc9955-5961-48d4-aca4-e225572d0efa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 21:02:12.976203 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:12.976169 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" event={"ID":"a2cd625d-00dd-459b-827a-510475c4b444","Type":"ContainerStarted","Data":"d118835d7832150d7bbc54a615732c66611aa86c27f1e0298e7a921c10c7cf94"} Oct 30 21:02:12.976393 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:12.976226 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" event={"ID":"a2cd625d-00dd-459b-827a-510475c4b444","Type":"ContainerStarted","Data":"9e25b33fa560c786b44b5633e4058bb8328d488faee1fcaa9ecfdad3faa8db67"} Oct 30 21:02:12.998961 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:12.998902 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" podStartSLOduration=1.998889327 podStartE2EDuration="1.998889327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:12.997039789 +0000 UTC m=+1438.176936311" watchObservedRunningTime="2026-10-30 21:02:12.998889327 +0000 UTC m=+1438.178785834" Oct 30 21:02:13.550100 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:13.550069 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:13.555918 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:13.555886 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:13.966369 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:13.966330 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 21:02:13.966585 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:13.966394 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" podUID="f5bc9955-5961-48d4-aca4-e225572d0efa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 21:02:13.983310 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:13.983280 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:13.984299 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:13.984275 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:14.973965 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:14.973934 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:02:14.975056 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:14.975030 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:02:15.999331 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:15.999299 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" Oct 30 21:02:18.201898 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:18.201849 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:18.206870 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:18.206830 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:02:18.950387 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:18.950334 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:19.061054 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:19.061003 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" podUID="14ba0389-1004-4f23-8560-cf6fe6f597dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.173:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:19.081920 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:19.080859 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:19.223308 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:19.223179 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:19.225152 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:19.225114 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": context deadline exceeded" Oct 30 21:02:19.273011 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:19.225861 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:02:19.273011 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:19.226372 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 21:02:19.273011 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:19.227575 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:19.273011 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:19.269180 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 21:02:19.965793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:19.961842 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:19.988980 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:19.984035 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 21:02:20.069682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:20.069625 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" podUID="14ba0389-1004-4f23-8560-cf6fe6f597dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.173:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:20.071587 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:20.071560 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:02:20.090694 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:20.087986 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 21:02:20.128689 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:20.128660 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 21:02:20.225979 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:20.225871 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:20.240109 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:20.240074 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 21:02:20.264898 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:20.241888 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 21:02:20.291794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:20.243041 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:02:20.670680 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:20.670603 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 21:02:20.809788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:20.809735 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 21:02:20.888648 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:20.875314 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 21:02:20.888648 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:20.875381 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 21:02:20.935185 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:20.932002 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 21:02:21.007163 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:21.006356 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:02:22.638310 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:02:22.638275 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:02:28.983758 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:28.983726 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48_eaaa8712-2a3c-42b8-897d-248d330bd675/payload-processing/0.log" Oct 30 21:02:29.186451 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:29.186420 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fztgr_bce11e0b-618d-40b0-9709-d6b3deda2012/payload-processing/0.log" Oct 30 21:02:29.392829 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:29.392721 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fztgr_bce11e0b-618d-40b0-9709-d6b3deda2012/payload-processing/0.log" Oct 30 21:02:29.583526 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:29.583497 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755_356460c8-2576-4249-8ba3-6efc7c78d124/payload-processing/0.log" Oct 30 21:02:29.787466 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:29.787434 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755_356460c8-2576-4249-8ba3-6efc7c78d124/payload-processing/0.log" Oct 30 21:02:29.787944 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:29.787921 4848 patch_prober.go:29] interesting pod/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:02:29.788132 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:29.788107 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" podUID="50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:31.121531 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.121497 4848 generic.go:356] "Generic (PLEG): container finished" podID="50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" containerID="1d8f8d4ca07607946b6bd623c37e21cceed8ec27b5bc91aea6e27c4d97e51d13" exitCode=0 Oct 30 21:02:31.122020 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.121571 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" event={"ID":"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8","Type":"ContainerDied","Data":"1d8f8d4ca07607946b6bd623c37e21cceed8ec27b5bc91aea6e27c4d97e51d13"} Oct 30 21:02:31.122177 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.122082 4848 scope.go:122] "RemoveContainer" containerID="1d8f8d4ca07607946b6bd623c37e21cceed8ec27b5bc91aea6e27c4d97e51d13" Oct 30 21:02:31.124013 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.123984 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:02:31.124100 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.124054 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:02:31.124100 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.124087 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:02:31.158390 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.158361 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p"] Oct 30 21:02:31.162573 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.162547 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" Oct 30 21:02:31.173244 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.173203 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-8fa15b-serving-cert\"" Oct 30 21:02:31.187206 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.187179 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p"] Oct 30 21:02:31.231183 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.231149 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p\" (UID: \"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" Oct 30 21:02:31.231337 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.231248 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td4h4\" (UniqueName: \"kubernetes.io/projected/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-kube-api-access-td4h4\") pod \"maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p\" (UID: \"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" Oct 30 21:02:31.231416 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.231384 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p\" (UID: \"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" Oct 30 21:02:31.231526 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.231482 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p\" (UID: \"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" Oct 30 21:02:31.355790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.339290 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p\" (UID: \"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" Oct 30 21:02:31.355790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.339483 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p\" (UID: \"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" Oct 30 21:02:31.355790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.339568 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td4h4\" (UniqueName: \"kubernetes.io/projected/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-kube-api-access-td4h4\") pod \"maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p\" (UID: \"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" Oct 30 21:02:31.355790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.339708 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p\" (UID: \"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" Oct 30 21:02:31.355790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.340627 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p\" (UID: \"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" Oct 30 21:02:31.361378 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.361342 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p\" (UID: \"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" Oct 30 21:02:31.398394 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.396957 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td4h4\" (UniqueName: \"kubernetes.io/projected/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-kube-api-access-td4h4\") pod \"maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p\" (UID: \"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" Oct 30 21:02:31.398394 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.397466 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p\" (UID: \"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" Oct 30 21:02:31.523424 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.523385 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" Oct 30 21:02:31.882807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:31.866502 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p"] Oct 30 21:02:32.148627 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:32.148533 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" event={"ID":"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8","Type":"ContainerStarted","Data":"9d59451822af795ebb75427a624b3e7dd8a75a466cdf648109eb1cc01e232139"} Oct 30 21:02:32.152477 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:32.152112 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" event={"ID":"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb","Type":"ContainerStarted","Data":"7af3b60ec852d29f64fa0ee67a77450735f48984ced9c6eedb4d25e4d2b5c8f0"} Oct 30 21:02:32.290776 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:32.290732 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6"] Oct 30 21:02:32.301206 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:32.301173 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv"] Oct 30 21:02:32.940817 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:32.939163 4848 patch_prober.go:29] interesting pod/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 21:02:32.940817 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:32.939223 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" podUID="50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 21:02:32.981789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:32.981744 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:02:33.161132 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:33.161084 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" event={"ID":"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb","Type":"ContainerStarted","Data":"4a2b6a9800e8db0fff01b274ddad8dd8ac8015eb89d295fc0338bf1f67f635f0"} Oct 30 21:02:33.161553 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:33.161395 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" Oct 30 21:02:33.179103 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:33.179028 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" podStartSLOduration=1.673523224 podStartE2EDuration="2.179013451s" totalImagesPullingTime="505.490227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:33.178295987 +0000 UTC m=+1458.358192494" watchObservedRunningTime="2026-10-30 21:02:33.179013451 +0000 UTC m=+1458.358909959" Oct 30 21:02:33.926792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:33.926259 4848 patch_prober.go:29] interesting pod/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 21:02:33.926792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:33.926320 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" podUID="50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 21:02:34.930012 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:34.929194 4848 patch_prober.go:29] interesting pod/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 21:02:34.930012 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:34.929252 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" podUID="50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 21:02:35.042620 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:35.041882 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq"] Oct 30 21:02:35.042620 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:35.042080 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" podUID="a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e" containerName="maas-api" containerID="cri-o://46ad8526774a685cd50ed188682d7aabbd09a7f33575e89a257c9c6f8a90cbf8" gracePeriod=30 Oct 30 21:02:35.178351 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:35.178299 4848 generic.go:356] "Generic (PLEG): container finished" podID="a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e" containerID="46ad8526774a685cd50ed188682d7aabbd09a7f33575e89a257c9c6f8a90cbf8" exitCode=0 Oct 30 21:02:35.178351 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:35.178350 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" event={"ID":"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e","Type":"ContainerDied","Data":"46ad8526774a685cd50ed188682d7aabbd09a7f33575e89a257c9c6f8a90cbf8"} Oct 30 21:02:35.638157 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:35.638091 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" Oct 30 21:02:35.751063 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:35.749849 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-metrics-certs\") pod \"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e\" (UID: \"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e\") " Oct 30 21:02:35.751063 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:35.749969 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-openshift-service-ca\") pod \"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e\" (UID: \"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e\") " Oct 30 21:02:35.751063 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:35.750018 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-maas-api-tls\") pod \"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e\" (UID: \"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e\") " Oct 30 21:02:35.751063 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:35.750047 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-kube-api-access-jt2n9\" (UniqueName: \"kubernetes.io/projected/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-kube-api-access-jt2n9\") pod \"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e\" (UID: \"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e\") " Oct 30 21:02:35.752280 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:35.752251 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-openshift-service-ca" pod "a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e" (UID: "a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:02:35.754361 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:35.754079 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-maas-api-tls" pod "a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e" (UID: "a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:02:35.754361 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:35.754174 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-kube-api-access-jt2n9" pod "a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e" (UID: "a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e"). InnerVolumeSpecName "kube-api-access-jt2n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:35.755202 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:35.755175 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-metrics-certs" pod "a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e" (UID: "a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:02:35.875182 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:35.858394 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:35.875453 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:35.875428 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:35.875563 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:35.875551 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:35.875653 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:35.875639 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt2n9\" (UniqueName: \"kubernetes.io/projected/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e-kube-api-access-jt2n9\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:35.920269 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:35.918730 4848 patch_prober.go:29] interesting pod/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 21:02:35.920269 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:35.918814 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" podUID="50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 21:02:36.187309 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:36.187276 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" Oct 30 21:02:36.187821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:36.187309 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq" event={"ID":"a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e","Type":"ContainerDied","Data":"39a08fc8479a8e6fa51c25147b4a9405c321b6aa37d8423b5c123a9665e63d19"} Oct 30 21:02:36.187821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:36.187372 4848 scope.go:122] "RemoveContainer" containerID="46ad8526774a685cd50ed188682d7aabbd09a7f33575e89a257c9c6f8a90cbf8" Oct 30 21:02:36.235619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:36.235578 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq"] Oct 30 21:02:36.239312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:36.239283 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-117ee1-574cd78db8-dhbkq"] Oct 30 21:02:36.928794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:36.927213 4848 patch_prober.go:29] interesting pod/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 21:02:36.928794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:36.927275 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" podUID="50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 21:02:36.938531 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:36.938495 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e" path="/var/lib/kubelet/pods/a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e/volumes" Oct 30 21:02:37.291619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:37.291531 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6" podUID="ea1a06fd-c761-43cb-b31c-47a22546106d" containerName="payload-processing" containerID="cri-o://ac58b98a77aaca23aa4a1d0faa6fa734634154d1be432fc13a8b9c5b787f4340" gracePeriod=25 Oct 30 21:02:37.302112 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:37.302069 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv" podUID="89b53a4d-165c-4b95-b0d2-913b50203b36" containerName="payload-pre-processing" containerID="cri-o://3ca477fd14641c45b4dbfef573adbfb96a3d44c7635a7b3bcd50d58ad8a8eb14" gracePeriod=25 Oct 30 21:02:37.753690 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:37.753659 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6" Oct 30 21:02:37.824563 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:37.824532 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv" Oct 30 21:02:37.858229 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:37.858197 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea1a06fd-c761-43cb-b31c-47a22546106d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea1a06fd-c761-43cb-b31c-47a22546106d-plugins-config-volume\") pod \"ea1a06fd-c761-43cb-b31c-47a22546106d\" (UID: \"ea1a06fd-c761-43cb-b31c-47a22546106d\") " Oct 30 21:02:37.858428 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:37.858244 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea1a06fd-c761-43cb-b31c-47a22546106d-kube-api-access-m9jp6\" (UniqueName: \"kubernetes.io/projected/ea1a06fd-c761-43cb-b31c-47a22546106d-kube-api-access-m9jp6\") pod \"ea1a06fd-c761-43cb-b31c-47a22546106d\" (UID: \"ea1a06fd-c761-43cb-b31c-47a22546106d\") " Oct 30 21:02:37.858719 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:37.858680 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea1a06fd-c761-43cb-b31c-47a22546106d-plugins-config-volume" pod "ea1a06fd-c761-43cb-b31c-47a22546106d" (UID: "ea1a06fd-c761-43cb-b31c-47a22546106d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:02:37.859020 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:37.858997 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea1a06fd-c761-43cb-b31c-47a22546106d-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:37.861170 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:37.861141 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea1a06fd-c761-43cb-b31c-47a22546106d-kube-api-access-m9jp6" pod "ea1a06fd-c761-43cb-b31c-47a22546106d" (UID: "ea1a06fd-c761-43cb-b31c-47a22546106d"). InnerVolumeSpecName "kube-api-access-m9jp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:37.910882 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:37.910830 4848 patch_prober.go:29] interesting pod/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 21:02:37.911100 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:37.910900 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" podUID="50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 21:02:37.959951 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:37.959905 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b53a4d-165c-4b95-b0d2-913b50203b36-kube-api-access-95hkv\" (UniqueName: \"kubernetes.io/projected/89b53a4d-165c-4b95-b0d2-913b50203b36-kube-api-access-95hkv\") pod \"89b53a4d-165c-4b95-b0d2-913b50203b36\" (UID: \"89b53a4d-165c-4b95-b0d2-913b50203b36\") " Oct 30 21:02:37.960159 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:37.960003 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b53a4d-165c-4b95-b0d2-913b50203b36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89b53a4d-165c-4b95-b0d2-913b50203b36-plugins-config-volume\") pod \"89b53a4d-165c-4b95-b0d2-913b50203b36\" (UID: \"89b53a4d-165c-4b95-b0d2-913b50203b36\") " Oct 30 21:02:37.960494 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:37.960462 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b53a4d-165c-4b95-b0d2-913b50203b36-plugins-config-volume" pod "89b53a4d-165c-4b95-b0d2-913b50203b36" (UID: "89b53a4d-165c-4b95-b0d2-913b50203b36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:02:37.960933 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:37.960912 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9jp6\" (UniqueName: \"kubernetes.io/projected/ea1a06fd-c761-43cb-b31c-47a22546106d-kube-api-access-m9jp6\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:37.960990 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:37.960940 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89b53a4d-165c-4b95-b0d2-913b50203b36-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:37.962215 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:37.962191 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b53a4d-165c-4b95-b0d2-913b50203b36-kube-api-access-95hkv" pod "89b53a4d-165c-4b95-b0d2-913b50203b36" (UID: "89b53a4d-165c-4b95-b0d2-913b50203b36"). InnerVolumeSpecName "kube-api-access-95hkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:38.062648 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.062616 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95hkv\" (UniqueName: \"kubernetes.io/projected/89b53a4d-165c-4b95-b0d2-913b50203b36-kube-api-access-95hkv\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:38.205458 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.205420 4848 generic.go:356] "Generic (PLEG): container finished" podID="ea1a06fd-c761-43cb-b31c-47a22546106d" containerID="ac58b98a77aaca23aa4a1d0faa6fa734634154d1be432fc13a8b9c5b787f4340" exitCode=0 Oct 30 21:02:38.205646 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.205468 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6" Oct 30 21:02:38.205646 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.205508 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6" event={"ID":"ea1a06fd-c761-43cb-b31c-47a22546106d","Type":"ContainerDied","Data":"ac58b98a77aaca23aa4a1d0faa6fa734634154d1be432fc13a8b9c5b787f4340"} Oct 30 21:02:38.205646 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.205579 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6" event={"ID":"ea1a06fd-c761-43cb-b31c-47a22546106d","Type":"ContainerDied","Data":"a26f6faba6ff3e75dcaa39189fb6674d609a0d2d5fff77b54f6df24f86d43e2f"} Oct 30 21:02:38.205646 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.205611 4848 scope.go:122] "RemoveContainer" containerID="ac58b98a77aaca23aa4a1d0faa6fa734634154d1be432fc13a8b9c5b787f4340" Oct 30 21:02:38.207500 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.207472 4848 generic.go:356] "Generic (PLEG): container finished" podID="89b53a4d-165c-4b95-b0d2-913b50203b36" containerID="3ca477fd14641c45b4dbfef573adbfb96a3d44c7635a7b3bcd50d58ad8a8eb14" exitCode=0 Oct 30 21:02:38.207615 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.207535 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv" Oct 30 21:02:38.207615 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.207540 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv" event={"ID":"89b53a4d-165c-4b95-b0d2-913b50203b36","Type":"ContainerDied","Data":"3ca477fd14641c45b4dbfef573adbfb96a3d44c7635a7b3bcd50d58ad8a8eb14"} Oct 30 21:02:38.207615 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.207591 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv" event={"ID":"89b53a4d-165c-4b95-b0d2-913b50203b36","Type":"ContainerDied","Data":"7bcd85333b569d83fcdb84afb60823d5601f491facc5723c5835a4518f4d6602"} Oct 30 21:02:38.222256 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.222229 4848 scope.go:122] "RemoveContainer" containerID="ac58b98a77aaca23aa4a1d0faa6fa734634154d1be432fc13a8b9c5b787f4340" Oct 30 21:02:38.222600 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:02:38.222580 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac58b98a77aaca23aa4a1d0faa6fa734634154d1be432fc13a8b9c5b787f4340\": container with ID starting with ac58b98a77aaca23aa4a1d0faa6fa734634154d1be432fc13a8b9c5b787f4340 not found: ID does not exist" containerID="ac58b98a77aaca23aa4a1d0faa6fa734634154d1be432fc13a8b9c5b787f4340" Oct 30 21:02:38.222690 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.222608 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac58b98a77aaca23aa4a1d0faa6fa734634154d1be432fc13a8b9c5b787f4340"} err="failed to get container status \"ac58b98a77aaca23aa4a1d0faa6fa734634154d1be432fc13a8b9c5b787f4340\": rpc error: code = NotFound desc = could not find container \"ac58b98a77aaca23aa4a1d0faa6fa734634154d1be432fc13a8b9c5b787f4340\": container with ID starting with ac58b98a77aaca23aa4a1d0faa6fa734634154d1be432fc13a8b9c5b787f4340 not found: ID does not exist" Oct 30 21:02:38.222690 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.222626 4848 scope.go:122] "RemoveContainer" containerID="3ca477fd14641c45b4dbfef573adbfb96a3d44c7635a7b3bcd50d58ad8a8eb14" Oct 30 21:02:38.233160 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.233132 4848 scope.go:122] "RemoveContainer" containerID="3ca477fd14641c45b4dbfef573adbfb96a3d44c7635a7b3bcd50d58ad8a8eb14" Oct 30 21:02:38.233446 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:02:38.233423 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca477fd14641c45b4dbfef573adbfb96a3d44c7635a7b3bcd50d58ad8a8eb14\": container with ID starting with 3ca477fd14641c45b4dbfef573adbfb96a3d44c7635a7b3bcd50d58ad8a8eb14 not found: ID does not exist" containerID="3ca477fd14641c45b4dbfef573adbfb96a3d44c7635a7b3bcd50d58ad8a8eb14" Oct 30 21:02:38.233518 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.233455 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca477fd14641c45b4dbfef573adbfb96a3d44c7635a7b3bcd50d58ad8a8eb14"} err="failed to get container status \"3ca477fd14641c45b4dbfef573adbfb96a3d44c7635a7b3bcd50d58ad8a8eb14\": rpc error: code = NotFound desc = could not find container \"3ca477fd14641c45b4dbfef573adbfb96a3d44c7635a7b3bcd50d58ad8a8eb14\": container with ID starting with 3ca477fd14641c45b4dbfef573adbfb96a3d44c7635a7b3bcd50d58ad8a8eb14 not found: ID does not exist" Oct 30 21:02:38.234949 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.234927 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6"] Oct 30 21:02:38.249168 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.249140 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-117ee1-84f85f57c5-h9xq6"] Oct 30 21:02:38.253507 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.253479 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s"] Oct 30 21:02:38.254183 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.254165 4848 cpu_manager.go:414] "containerMap: removing container" podUID="ea1a06fd-c761-43cb-b31c-47a22546106d" containerName="payload-processing" Oct 30 21:02:38.254284 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.254184 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea1a06fd-c761-43cb-b31c-47a22546106d" containerName="payload-processing" Oct 30 21:02:38.254284 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.254241 4848 cpu_manager.go:414] "containerMap: removing container" podUID="a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e" containerName="maas-api" Oct 30 21:02:38.254284 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.254246 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e" containerName="maas-api" Oct 30 21:02:38.254284 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.254259 4848 cpu_manager.go:414] "containerMap: removing container" podUID="89b53a4d-165c-4b95-b0d2-913b50203b36" containerName="payload-pre-processing" Oct 30 21:02:38.254284 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.254264 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b53a4d-165c-4b95-b0d2-913b50203b36" containerName="payload-pre-processing" Oct 30 21:02:38.254546 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.254382 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0f51d5e-d05f-4e44-b9f4-ddc7b36b7b1e" containerName="maas-api" Oct 30 21:02:38.254546 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.254390 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b53a4d-165c-4b95-b0d2-913b50203b36" containerName="payload-pre-processing" Oct 30 21:02:38.254546 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.254398 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea1a06fd-c761-43cb-b31c-47a22546106d" containerName="payload-processing" Oct 30 21:02:38.257719 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.257701 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" Oct 30 21:02:38.272330 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.272305 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-e53c37-serving-cert\"" Oct 30 21:02:38.285861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.285832 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s"] Oct 30 21:02:38.290306 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.290242 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv"] Oct 30 21:02:38.294559 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.294532 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-117ee1-7f4b7c479f-cnhzv"] Oct 30 21:02:38.396835 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.395137 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06664297-01f4-4a6f-80de-309ab01eabf0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s\" (UID: \"06664297-01f4-4a6f-80de-309ab01eabf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" Oct 30 21:02:38.396835 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.395242 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzffj\" (UniqueName: \"kubernetes.io/projected/06664297-01f4-4a6f-80de-309ab01eabf0-kube-api-access-nzffj\") pod \"maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s\" (UID: \"06664297-01f4-4a6f-80de-309ab01eabf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" Oct 30 21:02:38.396835 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.395358 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06664297-01f4-4a6f-80de-309ab01eabf0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s\" (UID: \"06664297-01f4-4a6f-80de-309ab01eabf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" Oct 30 21:02:38.396835 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.395398 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06664297-01f4-4a6f-80de-309ab01eabf0-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s\" (UID: \"06664297-01f4-4a6f-80de-309ab01eabf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" Oct 30 21:02:38.496878 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.496778 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06664297-01f4-4a6f-80de-309ab01eabf0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s\" (UID: \"06664297-01f4-4a6f-80de-309ab01eabf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" Oct 30 21:02:38.496878 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.496847 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06664297-01f4-4a6f-80de-309ab01eabf0-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s\" (UID: \"06664297-01f4-4a6f-80de-309ab01eabf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" Oct 30 21:02:38.497119 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.497025 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06664297-01f4-4a6f-80de-309ab01eabf0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s\" (UID: \"06664297-01f4-4a6f-80de-309ab01eabf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" Oct 30 21:02:38.497176 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.497126 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzffj\" (UniqueName: \"kubernetes.io/projected/06664297-01f4-4a6f-80de-309ab01eabf0-kube-api-access-nzffj\") pod \"maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s\" (UID: \"06664297-01f4-4a6f-80de-309ab01eabf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" Oct 30 21:02:38.498009 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.497980 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06664297-01f4-4a6f-80de-309ab01eabf0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s\" (UID: \"06664297-01f4-4a6f-80de-309ab01eabf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" Oct 30 21:02:38.499869 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.499847 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06664297-01f4-4a6f-80de-309ab01eabf0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s\" (UID: \"06664297-01f4-4a6f-80de-309ab01eabf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" Oct 30 21:02:38.500236 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.500217 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06664297-01f4-4a6f-80de-309ab01eabf0-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s\" (UID: \"06664297-01f4-4a6f-80de-309ab01eabf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" Oct 30 21:02:38.505452 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.505429 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzffj\" (UniqueName: \"kubernetes.io/projected/06664297-01f4-4a6f-80de-309ab01eabf0-kube-api-access-nzffj\") pod \"maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s\" (UID: \"06664297-01f4-4a6f-80de-309ab01eabf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" Oct 30 21:02:38.585928 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.585889 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" Oct 30 21:02:38.873059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:38.872977 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s"] Oct 30 21:02:38.877420 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:02:38.877394 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06664297_01f4_4a6f_80de_309ab01eabf0.slice/crio-355b6e9f16182c339be183b778f760df1e3cb0a5af74897ed6a4cab320725e9f WatchSource:0}: Error finding container 355b6e9f16182c339be183b778f760df1e3cb0a5af74897ed6a4cab320725e9f: Status 404 returned error can't find the container with id 355b6e9f16182c339be183b778f760df1e3cb0a5af74897ed6a4cab320725e9f Oct 30 21:02:39.016400 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:39.008576 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b53a4d-165c-4b95-b0d2-913b50203b36" path="/var/lib/kubelet/pods/89b53a4d-165c-4b95-b0d2-913b50203b36/volumes" Oct 30 21:02:39.016400 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:39.009227 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea1a06fd-c761-43cb-b31c-47a22546106d" path="/var/lib/kubelet/pods/ea1a06fd-c761-43cb-b31c-47a22546106d/volumes" Oct 30 21:02:39.038893 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:39.035885 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:02:39.038893 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:39.037750 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:02:39.045382 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:39.045359 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:02:39.174727 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:39.174706 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" Oct 30 21:02:39.223184 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:39.223150 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" event={"ID":"06664297-01f4-4a6f-80de-309ab01eabf0","Type":"ContainerStarted","Data":"355b6e9f16182c339be183b778f760df1e3cb0a5af74897ed6a4cab320725e9f"} Oct 30 21:02:40.107901 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.107865 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p"] Oct 30 21:02:40.112427 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.112403 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" Oct 30 21:02:40.128377 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.128336 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p"] Oct 30 21:02:40.128658 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.128641 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-74f9f4-serving-cert\"" Oct 30 21:02:40.258445 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.244257 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p\" (UID: \"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" Oct 30 21:02:40.258445 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.244324 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p\" (UID: \"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" Oct 30 21:02:40.258445 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.244547 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkg66\" (UniqueName: \"kubernetes.io/projected/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-kube-api-access-nkg66\") pod \"maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p\" (UID: \"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" Oct 30 21:02:40.258445 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.244633 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p\" (UID: \"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" Oct 30 21:02:40.322980 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.322943 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" event={"ID":"06664297-01f4-4a6f-80de-309ab01eabf0","Type":"ContainerStarted","Data":"80f026436581c1633fb51c8f4ba6ea2275c2a3f88fec700a901cb58c4f5005bd"} Oct 30 21:02:40.323320 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.323299 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" Oct 30 21:02:40.352806 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.352749 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p\" (UID: \"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" Oct 30 21:02:40.360161 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.360049 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p\" (UID: \"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" Oct 30 21:02:40.360161 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.360127 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p\" (UID: \"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" Oct 30 21:02:40.360536 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.360487 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkg66\" (UniqueName: \"kubernetes.io/projected/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-kube-api-access-nkg66\") pod \"maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p\" (UID: \"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" Oct 30 21:02:40.362569 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.354594 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p\" (UID: \"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" Oct 30 21:02:40.363263 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.363206 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" podStartSLOduration=1.7572006249999998 podStartE2EDuration="2.363189176s" totalImagesPullingTime="605.988551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:40.356161748 +0000 UTC m=+1465.536058256" watchObservedRunningTime="2026-10-30 21:02:40.363189176 +0000 UTC m=+1465.543085688" Oct 30 21:02:40.369808 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.366242 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p\" (UID: \"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" Oct 30 21:02:40.369808 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.368159 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p\" (UID: \"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" Oct 30 21:02:40.371202 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.371181 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkg66\" (UniqueName: \"kubernetes.io/projected/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-kube-api-access-nkg66\") pod \"maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p\" (UID: \"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" Oct 30 21:02:40.443599 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.443559 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" Oct 30 21:02:40.865196 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:40.865147 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p"] Oct 30 21:02:41.328497 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:41.328463 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" event={"ID":"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f","Type":"ContainerStarted","Data":"65c6fa491b226174e4c4303da645a0b7eb4e70e88e94b6a3f80e78a24b67f975"} Oct 30 21:02:42.342356 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:42.342303 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" event={"ID":"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f","Type":"ContainerStarted","Data":"3fa7123a118cfac148090652202418fcf0a0cc15402776318d69943aadda9862"} Oct 30 21:02:42.342919 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:42.342547 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" Oct 30 21:02:42.369450 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:42.369369 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" podStartSLOduration=1.922908683 podStartE2EDuration="2.369348353s" totalImagesPullingTime="446.43967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:42.364363795 +0000 UTC m=+1467.544260303" watchObservedRunningTime="2026-10-30 21:02:42.369348353 +0000 UTC m=+1467.549244861" Oct 30 21:02:44.354340 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:44.340426 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g"] Oct 30 21:02:44.436195 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:44.411967 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" podUID="f5bc9955-5961-48d4-aca4-e225572d0efa" containerName="istio-proxy" containerID="cri-o://92824b159b6419959779f749ad35eaad08ebfc6015fbbeac7be58f431c6c7e48" gracePeriod=30 Oct 30 21:02:44.872425 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:44.872380 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 21:02:44.957792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:44.949307 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:45.215786 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:45.210838 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 21:02:45.215786 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:45.210905 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" podUID="f5bc9955-5961-48d4-aca4-e225572d0efa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:02:45.302915 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:45.302868 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:45.360995 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:45.356879 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:02:45.360995 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:45.359484 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" podUID="f7cbc43b-dad3-4ab6-ad12-e03522a725b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:45.798368 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:45.741733 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 21:02:45.814918 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:45.814872 4848 patch_prober.go:29] interesting pod/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:02:45.815191 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:45.815158 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" podUID="8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:45.955800 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:45.947200 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:46.047795 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:45.980859 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:46.047795 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:46.045062 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 21:02:46.047795 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:46.045123 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 21:02:46.047795 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:45.984715 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" podUID="14ba0389-1004-4f23-8560-cf6fe6f597dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.173:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:46.304819 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:46.291971 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:46.428046 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:46.427993 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:46.605844 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:46.436662 4848 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" podUID="14ba0389-1004-4f23-8560-cf6fe6f597dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.173:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:02:46.701089 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:46.665834 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 21:02:46.939171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:46.939147 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 21:02:46.992295 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:46.992265 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 21:02:46.992544 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:46.992528 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 21:02:47.350916 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:47.350843 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" Oct 30 21:02:48.386447 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:48.386414 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" Oct 30 21:02:49.734514 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.734450 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:02:49.830429 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.830383 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-credential-socket\") pod \"f5bc9955-5961-48d4-aca4-e225572d0efa\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " Oct 30 21:02:49.830608 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.830442 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-data\") pod \"f5bc9955-5961-48d4-aca4-e225572d0efa\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " Oct 30 21:02:49.830608 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.830486 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-workload-certs\") pod \"f5bc9955-5961-48d4-aca4-e225572d0efa\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " Oct 30 21:02:49.830608 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.830569 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-workload-socket\") pod \"f5bc9955-5961-48d4-aca4-e225572d0efa\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " Oct 30 21:02:49.830871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.830645 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-envoy\") pod \"f5bc9955-5961-48d4-aca4-e225572d0efa\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " Oct 30 21:02:49.830871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.830685 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-token\" (UniqueName: \"kubernetes.io/projected/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-token\") pod \"f5bc9955-5961-48d4-aca4-e225572d0efa\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " Oct 30 21:02:49.830871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.830743 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5bc9955-5961-48d4-aca4-e225572d0efa-kube-api-access-5kzp2\" (UniqueName: \"kubernetes.io/projected/f5bc9955-5961-48d4-aca4-e225572d0efa-kube-api-access-5kzp2\") pod \"f5bc9955-5961-48d4-aca4-e225572d0efa\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " Oct 30 21:02:49.830871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.830859 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-podinfo\") pod \"f5bc9955-5961-48d4-aca4-e225572d0efa\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " Oct 30 21:02:49.831101 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.831028 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5bc9955-5961-48d4-aca4-e225572d0efa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5bc9955-5961-48d4-aca4-e225572d0efa-istiod-ca-cert\") pod \"f5bc9955-5961-48d4-aca4-e225572d0efa\" (UID: \"f5bc9955-5961-48d4-aca4-e225572d0efa\") " Oct 30 21:02:49.831343 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.831312 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-workload-socket" pod "f5bc9955-5961-48d4-aca4-e225572d0efa" (UID: "f5bc9955-5961-48d4-aca4-e225572d0efa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:49.831729 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.831705 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:49.831849 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.831721 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5bc9955-5961-48d4-aca4-e225572d0efa-istiod-ca-cert" pod "f5bc9955-5961-48d4-aca4-e225572d0efa" (UID: "f5bc9955-5961-48d4-aca4-e225572d0efa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:02:49.831944 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.831922 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-credential-socket" pod "f5bc9955-5961-48d4-aca4-e225572d0efa" (UID: "f5bc9955-5961-48d4-aca4-e225572d0efa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:49.832372 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.832116 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-workload-certs" pod "f5bc9955-5961-48d4-aca4-e225572d0efa" (UID: "f5bc9955-5961-48d4-aca4-e225572d0efa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:49.834684 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.834641 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-podinfo" pod "f5bc9955-5961-48d4-aca4-e225572d0efa" (UID: "f5bc9955-5961-48d4-aca4-e225572d0efa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:02:49.834684 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.834673 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5bc9955-5961-48d4-aca4-e225572d0efa-kube-api-access-5kzp2" pod "f5bc9955-5961-48d4-aca4-e225572d0efa" (UID: "f5bc9955-5961-48d4-aca4-e225572d0efa"). InnerVolumeSpecName "kube-api-access-5kzp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:49.835056 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.835036 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-data" pod "f5bc9955-5961-48d4-aca4-e225572d0efa" (UID: "f5bc9955-5961-48d4-aca4-e225572d0efa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:49.835307 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.835282 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-token" pod "f5bc9955-5961-48d4-aca4-e225572d0efa" (UID: "f5bc9955-5961-48d4-aca4-e225572d0efa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:02:49.836303 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.836280 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-envoy" pod "f5bc9955-5961-48d4-aca4-e225572d0efa" (UID: "f5bc9955-5961-48d4-aca4-e225572d0efa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:02:49.932860 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.932821 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5bc9955-5961-48d4-aca4-e225572d0efa-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:49.932860 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.932853 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:49.932860 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.932863 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:49.932860 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.932871 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:49.933239 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.932880 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:49.933239 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.932889 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:49.933239 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.932898 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kzp2\" (UniqueName: \"kubernetes.io/projected/f5bc9955-5961-48d4-aca4-e225572d0efa-kube-api-access-5kzp2\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:49.933239 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:49.932908 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5bc9955-5961-48d4-aca4-e225572d0efa-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:02:50.462787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:50.462520 4848 generic.go:356] "Generic (PLEG): container finished" podID="f5bc9955-5961-48d4-aca4-e225572d0efa" containerID="92824b159b6419959779f749ad35eaad08ebfc6015fbbeac7be58f431c6c7e48" exitCode=0 Oct 30 21:02:50.462977 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:50.462818 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" event={"ID":"f5bc9955-5961-48d4-aca4-e225572d0efa","Type":"ContainerDied","Data":"92824b159b6419959779f749ad35eaad08ebfc6015fbbeac7be58f431c6c7e48"} Oct 30 21:02:50.462977 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:50.462881 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" event={"ID":"f5bc9955-5961-48d4-aca4-e225572d0efa","Type":"ContainerDied","Data":"af0a0be4a7680894ff75caebafd46279d019edfb3b14c43368e4f9c6f1500581"} Oct 30 21:02:50.462977 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:50.462912 4848 scope.go:122] "RemoveContainer" containerID="92824b159b6419959779f749ad35eaad08ebfc6015fbbeac7be58f431c6c7e48" Oct 30 21:02:50.463180 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:50.463119 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g" Oct 30 21:02:50.514395 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:50.514369 4848 scope.go:122] "RemoveContainer" containerID="24a742a7fc2d50572a9fca0826f29b959e56c3d96a6fbfa3b4fc752392c3af41" Oct 30 21:02:50.532065 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:50.531732 4848 scope.go:122] "RemoveContainer" containerID="92824b159b6419959779f749ad35eaad08ebfc6015fbbeac7be58f431c6c7e48" Oct 30 21:02:50.532619 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:02:50.532275 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92824b159b6419959779f749ad35eaad08ebfc6015fbbeac7be58f431c6c7e48\": container with ID starting with 92824b159b6419959779f749ad35eaad08ebfc6015fbbeac7be58f431c6c7e48 not found: ID does not exist" containerID="92824b159b6419959779f749ad35eaad08ebfc6015fbbeac7be58f431c6c7e48" Oct 30 21:02:50.532619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:50.532316 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92824b159b6419959779f749ad35eaad08ebfc6015fbbeac7be58f431c6c7e48"} err="failed to get container status \"92824b159b6419959779f749ad35eaad08ebfc6015fbbeac7be58f431c6c7e48\": rpc error: code = NotFound desc = could not find container \"92824b159b6419959779f749ad35eaad08ebfc6015fbbeac7be58f431c6c7e48\": container with ID starting with 92824b159b6419959779f749ad35eaad08ebfc6015fbbeac7be58f431c6c7e48 not found: ID does not exist" Oct 30 21:02:50.532619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:50.532353 4848 scope.go:122] "RemoveContainer" containerID="24a742a7fc2d50572a9fca0826f29b959e56c3d96a6fbfa3b4fc752392c3af41" Oct 30 21:02:50.533440 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:02:50.533353 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a742a7fc2d50572a9fca0826f29b959e56c3d96a6fbfa3b4fc752392c3af41\": container with ID starting with 24a742a7fc2d50572a9fca0826f29b959e56c3d96a6fbfa3b4fc752392c3af41 not found: ID does not exist" containerID="24a742a7fc2d50572a9fca0826f29b959e56c3d96a6fbfa3b4fc752392c3af41" Oct 30 21:02:50.533440 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:50.533386 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a742a7fc2d50572a9fca0826f29b959e56c3d96a6fbfa3b4fc752392c3af41"} err="failed to get container status \"24a742a7fc2d50572a9fca0826f29b959e56c3d96a6fbfa3b4fc752392c3af41\": rpc error: code = NotFound desc = could not find container \"24a742a7fc2d50572a9fca0826f29b959e56c3d96a6fbfa3b4fc752392c3af41\": container with ID starting with 24a742a7fc2d50572a9fca0826f29b959e56c3d96a6fbfa3b4fc752392c3af41 not found: ID does not exist" Oct 30 21:02:50.541361 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:50.541311 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g"] Oct 30 21:02:50.544955 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:50.544932 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-117ee1-openshift-default-79d78bbff7-tg64g"] Oct 30 21:02:50.968310 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:50.968275 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5bc9955-5961-48d4-aca4-e225572d0efa" path="/var/lib/kubelet/pods/f5bc9955-5961-48d4-aca4-e225572d0efa/volumes" Oct 30 21:02:51.702269 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.702210 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7"] Oct 30 21:02:51.703267 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.703237 4848 cpu_manager.go:414] "containerMap: removing container" podUID="f5bc9955-5961-48d4-aca4-e225572d0efa" containerName="istio-proxy" Oct 30 21:02:51.703267 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.703258 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5bc9955-5961-48d4-aca4-e225572d0efa" containerName="istio-proxy" Oct 30 21:02:51.703519 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.703300 4848 cpu_manager.go:414] "containerMap: removing container" podUID="f5bc9955-5961-48d4-aca4-e225572d0efa" containerName="istio-proxy" Oct 30 21:02:51.703519 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.703309 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5bc9955-5961-48d4-aca4-e225572d0efa" containerName="istio-proxy" Oct 30 21:02:51.703519 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.703495 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5bc9955-5961-48d4-aca4-e225572d0efa" containerName="istio-proxy" Oct 30 21:02:51.707968 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.707936 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.715500 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.715477 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-f64640-openshift-default-dockercfg-lqkjk\"" Oct 30 21:02:51.723010 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.722974 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7"] Oct 30 21:02:51.775114 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.775075 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlv6n\" (UniqueName: \"kubernetes.io/projected/7f1686ee-2109-4bfa-9964-cc40955a35b9-kube-api-access-rlv6n\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.775293 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.775169 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-token\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.775293 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.775206 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f1686ee-2109-4bfa-9964-cc40955a35b9-istiod-ca-cert\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.775384 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.775297 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-workload-certs\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.775384 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.775349 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-workload-socket\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.775566 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.775546 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-credential-socket\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.775704 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.775686 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-data\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.775792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.775722 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-podinfo\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.775792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.775756 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-envoy\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.892239 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.892195 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-credential-socket\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.896788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.893934 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-data\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.896788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.893984 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-podinfo\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.896788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.894019 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-envoy\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.896788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.894146 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlv6n\" (UniqueName: \"kubernetes.io/projected/7f1686ee-2109-4bfa-9964-cc40955a35b9-kube-api-access-rlv6n\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.896788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.894182 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-token\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.896788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.894209 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f1686ee-2109-4bfa-9964-cc40955a35b9-istiod-ca-cert\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.896788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.894245 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-workload-certs\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.896788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.894292 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-workload-socket\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.896788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.894789 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-workload-socket\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.896788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.892929 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-credential-socket\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.896788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.895054 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-data\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.901789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.898288 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f1686ee-2109-4bfa-9964-cc40955a35b9-istiod-ca-cert\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.902005 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.898965 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-workload-certs\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.948792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.937298 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-token\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:51.960951 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:51.960741 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlv6n\" (UniqueName: \"kubernetes.io/projected/7f1686ee-2109-4bfa-9964-cc40955a35b9-kube-api-access-rlv6n\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:52.074175 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:52.074144 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-envoy\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:52.075600 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:52.075574 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-podinfo\") pod \"e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:52.329723 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:52.329627 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:52.536670 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:02:52.536628 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:02:52.649324 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:52.649293 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7"] Oct 30 21:02:52.652570 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:52.652536 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:02:52.652646 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:52.652618 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:02:52.652682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:52.652657 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:02:53.514090 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:53.514048 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" event={"ID":"7f1686ee-2109-4bfa-9964-cc40955a35b9","Type":"ContainerStarted","Data":"794183a3dee72e74d092a786187b0bc7b8acf59556b005b53f7359353a405e26"} Oct 30 21:02:53.514631 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:53.514146 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" event={"ID":"7f1686ee-2109-4bfa-9964-cc40955a35b9","Type":"ContainerStarted","Data":"ce5b94087b4c788148082cedab2637342b93abc1c060ced091e7f188ac1a9e35"} Oct 30 21:02:53.546418 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:53.546325 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" podStartSLOduration=2.546296504 podStartE2EDuration="2.546296504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:53.542292965 +0000 UTC m=+1478.722189472" watchObservedRunningTime="2026-10-30 21:02:53.546296504 +0000 UTC m=+1478.726193013" Oct 30 21:02:54.330021 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:54.329986 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:54.336575 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:54.336551 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:54.522733 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:54.522702 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:54.523694 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:54.523673 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:02:55.087968 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:55.087925 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx"] Oct 30 21:02:55.104367 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:55.104328 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc"] Oct 30 21:02:56.575007 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:56.574969 4848 generic.go:356] "Generic (PLEG): container finished" podID="a2cd625d-00dd-459b-827a-510475c4b444" containerID="d118835d7832150d7bbc54a615732c66611aa86c27f1e0298e7a921c10c7cf94" exitCode=0 Oct 30 21:02:56.585824 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:56.577828 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" event={"ID":"a2cd625d-00dd-459b-827a-510475c4b444","Type":"ContainerDied","Data":"d118835d7832150d7bbc54a615732c66611aa86c27f1e0298e7a921c10c7cf94"} Oct 30 21:02:56.586367 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:56.586338 4848 scope.go:122] "RemoveContainer" containerID="d118835d7832150d7bbc54a615732c66611aa86c27f1e0298e7a921c10c7cf94" Oct 30 21:02:56.589410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:56.589367 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:02:56.589527 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:56.589469 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:02:56.589527 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:56.589519 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:02:57.075460 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.063004 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq"] Oct 30 21:02:57.075460 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.064665 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5bc9955-5961-48d4-aca4-e225572d0efa" containerName="istio-proxy" Oct 30 21:02:57.075460 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.074092 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.082172 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.080576 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-321ccd-openshift-default-dockercfg-f2b7f\"" Oct 30 21:02:57.082172 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.081456 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq"] Oct 30 21:02:57.171203 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.171162 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-workload-certs\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.171381 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.171270 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a47632-b985-464f-8ee1-4fe60a1b3258-istiod-ca-cert\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.171381 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.171299 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-envoy\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.171381 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.171343 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngpmh\" (UniqueName: \"kubernetes.io/projected/01a47632-b985-464f-8ee1-4fe60a1b3258-kube-api-access-ngpmh\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.171381 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.171372 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-token\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.171651 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.171493 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-data\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.171651 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.171562 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-podinfo\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.171808 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.171671 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-credential-socket\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.171863 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.171814 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-workload-socket\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.346894 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.346801 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-workload-certs\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.365794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.357041 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-workload-certs\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.365794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.364521 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a47632-b985-464f-8ee1-4fe60a1b3258-istiod-ca-cert\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.366999 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.366970 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a47632-b985-464f-8ee1-4fe60a1b3258-istiod-ca-cert\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.367283 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.367264 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-envoy\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.371946 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.371918 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngpmh\" (UniqueName: \"kubernetes.io/projected/01a47632-b985-464f-8ee1-4fe60a1b3258-kube-api-access-ngpmh\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.372720 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.372686 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-token\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.384790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.384745 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-data\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.386503 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.385098 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-data\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.388095 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.388050 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-envoy\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.394078 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.394054 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-podinfo\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.394734 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.394713 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-credential-socket\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.395009 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.394979 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-workload-socket\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.395554 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.395534 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-workload-socket\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.395976 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.395957 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-credential-socket\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.486371 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.485278 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngpmh\" (UniqueName: \"kubernetes.io/projected/01a47632-b985-464f-8ee1-4fe60a1b3258-kube-api-access-ngpmh\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.512787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.495084 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-podinfo\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.512787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.495527 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-token\") pod \"e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.704130 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.704100 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:57.944979 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:57.944939 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" event={"ID":"a2cd625d-00dd-459b-827a-510475c4b444","Type":"ContainerStarted","Data":"30c5592f723509b1052226ea392a6dbde72828e28da759233f7ad84866015eef"} Oct 30 21:02:58.233368 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:58.233294 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq"] Oct 30 21:02:58.233923 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:02:58.233891 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a47632_b985_464f_8ee1_4fe60a1b3258.slice/crio-ef4ae02c98e099252291635ed6eeed19452779c14d4a53330d8a61f18562398b WatchSource:0}: Error finding container ef4ae02c98e099252291635ed6eeed19452779c14d4a53330d8a61f18562398b: Status 404 returned error can't find the container with id ef4ae02c98e099252291635ed6eeed19452779c14d4a53330d8a61f18562398b Oct 30 21:02:58.236569 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:58.236536 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:02:58.236648 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:58.236604 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:02:58.236648 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:58.236636 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:02:58.556837 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:58.554675 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:58.557024 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:58.556915 4848 patch_prober.go:29] interesting pod/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 21:02:58.557024 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:58.556969 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" podUID="a2cd625d-00dd-459b-827a-510475c4b444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 21:02:58.953215 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:58.953185 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" event={"ID":"01a47632-b985-464f-8ee1-4fe60a1b3258","Type":"ContainerStarted","Data":"6adfe6528c1330c09fabb1d8b4ab1f784850138591c7147430f5c3f18fba47f5"} Oct 30 21:02:58.953646 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:58.953240 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" event={"ID":"01a47632-b985-464f-8ee1-4fe60a1b3258","Type":"ContainerStarted","Data":"ef4ae02c98e099252291635ed6eeed19452779c14d4a53330d8a61f18562398b"} Oct 30 21:02:58.975785 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:58.975709 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" podStartSLOduration=1.9756910890000001 podStartE2EDuration="1.975691089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:02:58.972592565 +0000 UTC m=+1484.152489072" watchObservedRunningTime="2026-10-30 21:02:58.975691089 +0000 UTC m=+1484.155587598" Oct 30 21:02:58.983608 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:58.983577 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:02:59.550745 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:59.550711 4848 patch_prober.go:29] interesting pod/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 21:02:59.550998 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:59.550760 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" podUID="a2cd625d-00dd-459b-827a-510475c4b444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 21:02:59.731454 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:59.714109 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:59.731454 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:59.731059 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:59.961030 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:59.960996 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:02:59.962205 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:02:59.962183 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:03:00.091798 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:00.089441 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx" podUID="cc8792ce-9e3b-47c0-93a5-e5a7ded65612" containerName="payload-processing" containerID="cri-o://755b84fdf420ec8cbc20670ce5014fe6244976998d919e9a5ca3aeb21c6f660f" gracePeriod=25 Oct 30 21:03:00.165975 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:00.159563 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc" podUID="5dd5f82e-2c41-4971-ad3b-2bb874ae225c" containerName="payload-pre-processing" containerID="cri-o://1aff8a5fe1e76dfc9e1741f273b325000ade82312c31f1e07f1e121fae3d2fba" gracePeriod=25 Oct 30 21:03:00.356483 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:03:00.356449 4848 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd5f82e_2c41_4971_ad3b_2bb874ae225c.slice/crio-conmon-1aff8a5fe1e76dfc9e1741f273b325000ade82312c31f1e07f1e121fae3d2fba.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:03:00.550177 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:00.550144 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx" Oct 30 21:03:00.550491 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:00.550461 4848 patch_prober.go:29] interesting pod/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 21:03:00.550654 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:00.550515 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" podUID="a2cd625d-00dd-459b-827a-510475c4b444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 21:03:00.640619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:00.640592 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc" Oct 30 21:03:00.669437 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:00.669405 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8792ce-9e3b-47c0-93a5-e5a7ded65612-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8792ce-9e3b-47c0-93a5-e5a7ded65612-plugins-config-volume\") pod \"cc8792ce-9e3b-47c0-93a5-e5a7ded65612\" (UID: \"cc8792ce-9e3b-47c0-93a5-e5a7ded65612\") " Oct 30 21:03:00.669672 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:00.669495 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8792ce-9e3b-47c0-93a5-e5a7ded65612-kube-api-access-95jw4\" (UniqueName: \"kubernetes.io/projected/cc8792ce-9e3b-47c0-93a5-e5a7ded65612-kube-api-access-95jw4\") pod \"cc8792ce-9e3b-47c0-93a5-e5a7ded65612\" (UID: \"cc8792ce-9e3b-47c0-93a5-e5a7ded65612\") " Oct 30 21:03:00.669940 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:00.669913 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8792ce-9e3b-47c0-93a5-e5a7ded65612-plugins-config-volume" pod "cc8792ce-9e3b-47c0-93a5-e5a7ded65612" (UID: "cc8792ce-9e3b-47c0-93a5-e5a7ded65612"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:03:00.670260 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:00.670236 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc8792ce-9e3b-47c0-93a5-e5a7ded65612-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:00.672174 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:00.672147 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8792ce-9e3b-47c0-93a5-e5a7ded65612-kube-api-access-95jw4" pod "cc8792ce-9e3b-47c0-93a5-e5a7ded65612" (UID: "cc8792ce-9e3b-47c0-93a5-e5a7ded65612"). InnerVolumeSpecName "kube-api-access-95jw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:03:00.778486 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:00.778438 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dd5f82e-2c41-4971-ad3b-2bb874ae225c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd5f82e-2c41-4971-ad3b-2bb874ae225c-plugins-config-volume\") pod \"5dd5f82e-2c41-4971-ad3b-2bb874ae225c\" (UID: \"5dd5f82e-2c41-4971-ad3b-2bb874ae225c\") " Oct 30 21:03:00.778693 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:00.778628 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd5f82e-2c41-4971-ad3b-2bb874ae225c-kube-api-access-9ntv5\" (UniqueName: \"kubernetes.io/projected/5dd5f82e-2c41-4971-ad3b-2bb874ae225c-kube-api-access-9ntv5\") pod \"5dd5f82e-2c41-4971-ad3b-2bb874ae225c\" (UID: \"5dd5f82e-2c41-4971-ad3b-2bb874ae225c\") " Oct 30 21:03:00.785470 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:00.785431 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd5f82e-2c41-4971-ad3b-2bb874ae225c-plugins-config-volume" pod "5dd5f82e-2c41-4971-ad3b-2bb874ae225c" (UID: "5dd5f82e-2c41-4971-ad3b-2bb874ae225c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:03:00.819840 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:00.801367 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dd5f82e-2c41-4971-ad3b-2bb874ae225c-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:00.820359 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:00.820321 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95jw4\" (UniqueName: \"kubernetes.io/projected/cc8792ce-9e3b-47c0-93a5-e5a7ded65612-kube-api-access-95jw4\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:00.898802 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:00.874570 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd5f82e-2c41-4971-ad3b-2bb874ae225c-kube-api-access-9ntv5" pod "5dd5f82e-2c41-4971-ad3b-2bb874ae225c" (UID: "5dd5f82e-2c41-4971-ad3b-2bb874ae225c"). InnerVolumeSpecName "kube-api-access-9ntv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:03:00.939799 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:00.921861 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ntv5\" (UniqueName: \"kubernetes.io/projected/5dd5f82e-2c41-4971-ad3b-2bb874ae225c-kube-api-access-9ntv5\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:01.159539 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:01.158557 4848 generic.go:356] "Generic (PLEG): container finished" podID="5dd5f82e-2c41-4971-ad3b-2bb874ae225c" containerID="1aff8a5fe1e76dfc9e1741f273b325000ade82312c31f1e07f1e121fae3d2fba" exitCode=0 Oct 30 21:03:01.159539 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:01.158947 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc" event={"ID":"5dd5f82e-2c41-4971-ad3b-2bb874ae225c","Type":"ContainerDied","Data":"1aff8a5fe1e76dfc9e1741f273b325000ade82312c31f1e07f1e121fae3d2fba"} Oct 30 21:03:01.159539 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:01.159021 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc" event={"ID":"5dd5f82e-2c41-4971-ad3b-2bb874ae225c","Type":"ContainerDied","Data":"dea3e64d7ab1b5dd58d43435bc839622b7e39391302ebb4fcc798714b370a6a2"} Oct 30 21:03:01.159539 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:01.159055 4848 scope.go:122] "RemoveContainer" containerID="1aff8a5fe1e76dfc9e1741f273b325000ade82312c31f1e07f1e121fae3d2fba" Oct 30 21:03:01.159539 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:01.159271 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc" Oct 30 21:03:01.165645 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:01.165612 4848 generic.go:356] "Generic (PLEG): container finished" podID="cc8792ce-9e3b-47c0-93a5-e5a7ded65612" containerID="755b84fdf420ec8cbc20670ce5014fe6244976998d919e9a5ca3aeb21c6f660f" exitCode=0 Oct 30 21:03:01.165930 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:01.165913 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx" Oct 30 21:03:01.166265 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:01.166242 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx" event={"ID":"cc8792ce-9e3b-47c0-93a5-e5a7ded65612","Type":"ContainerDied","Data":"755b84fdf420ec8cbc20670ce5014fe6244976998d919e9a5ca3aeb21c6f660f"} Oct 30 21:03:01.166349 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:01.166292 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx" event={"ID":"cc8792ce-9e3b-47c0-93a5-e5a7ded65612","Type":"ContainerDied","Data":"59a06d7304d3ab305cb5525e4897874ed7220158e91415efdc30d2a2049252bd"} Oct 30 21:03:01.181368 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:01.181350 4848 scope.go:122] "RemoveContainer" containerID="1aff8a5fe1e76dfc9e1741f273b325000ade82312c31f1e07f1e121fae3d2fba" Oct 30 21:03:01.181670 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:03:01.181649 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aff8a5fe1e76dfc9e1741f273b325000ade82312c31f1e07f1e121fae3d2fba\": container with ID starting with 1aff8a5fe1e76dfc9e1741f273b325000ade82312c31f1e07f1e121fae3d2fba not found: ID does not exist" containerID="1aff8a5fe1e76dfc9e1741f273b325000ade82312c31f1e07f1e121fae3d2fba" Oct 30 21:03:01.181715 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:01.181678 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aff8a5fe1e76dfc9e1741f273b325000ade82312c31f1e07f1e121fae3d2fba"} err="failed to get container status \"1aff8a5fe1e76dfc9e1741f273b325000ade82312c31f1e07f1e121fae3d2fba\": rpc error: code = NotFound desc = could not find container \"1aff8a5fe1e76dfc9e1741f273b325000ade82312c31f1e07f1e121fae3d2fba\": container with ID starting with 1aff8a5fe1e76dfc9e1741f273b325000ade82312c31f1e07f1e121fae3d2fba not found: ID does not exist" Oct 30 21:03:01.181715 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:01.181695 4848 scope.go:122] "RemoveContainer" containerID="755b84fdf420ec8cbc20670ce5014fe6244976998d919e9a5ca3aeb21c6f660f" Oct 30 21:03:01.192689 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:01.192652 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx"] Oct 30 21:03:01.197142 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:01.197120 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fa48f3-5d4c4445d9-njphx"] Oct 30 21:03:01.295812 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:01.293862 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc"] Oct 30 21:03:01.390796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:01.387369 4848 scope.go:122] "RemoveContainer" containerID="755b84fdf420ec8cbc20670ce5014fe6244976998d919e9a5ca3aeb21c6f660f" Oct 30 21:03:01.390796 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:03:01.389657 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"755b84fdf420ec8cbc20670ce5014fe6244976998d919e9a5ca3aeb21c6f660f\": container with ID starting with 755b84fdf420ec8cbc20670ce5014fe6244976998d919e9a5ca3aeb21c6f660f not found: ID does not exist" containerID="755b84fdf420ec8cbc20670ce5014fe6244976998d919e9a5ca3aeb21c6f660f" Oct 30 21:03:01.390796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:01.389701 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"755b84fdf420ec8cbc20670ce5014fe6244976998d919e9a5ca3aeb21c6f660f"} err="failed to get container status \"755b84fdf420ec8cbc20670ce5014fe6244976998d919e9a5ca3aeb21c6f660f\": rpc error: code = NotFound desc = could not find container \"755b84fdf420ec8cbc20670ce5014fe6244976998d919e9a5ca3aeb21c6f660f\": container with ID starting with 755b84fdf420ec8cbc20670ce5014fe6244976998d919e9a5ca3aeb21c6f660f not found: ID does not exist" Oct 30 21:03:01.414849 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:01.414463 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fa48f3-c5f9f5c7f-lx8tc"] Oct 30 21:03:01.551599 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:01.551021 4848 patch_prober.go:29] interesting pod/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 21:03:01.551599 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:01.551104 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" podUID="a2cd625d-00dd-459b-827a-510475c4b444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 21:03:02.549932 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:02.549896 4848 patch_prober.go:29] interesting pod/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 21:03:02.550373 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:02.549952 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" podUID="a2cd625d-00dd-459b-827a-510475c4b444" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 21:03:02.940218 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:02.940187 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dd5f82e-2c41-4971-ad3b-2bb874ae225c" path="/var/lib/kubelet/pods/5dd5f82e-2c41-4971-ad3b-2bb874ae225c/volumes" Oct 30 21:03:02.940684 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:02.940667 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc8792ce-9e3b-47c0-93a5-e5a7ded65612" path="/var/lib/kubelet/pods/cc8792ce-9e3b-47c0-93a5-e5a7ded65612/volumes" Oct 30 21:03:03.415279 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:03.415191 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48"] Oct 30 21:03:03.415461 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:03.415409 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48" podUID="eaaa8712-2a3c-42b8-897d-248d330bd675" containerName="payload-processing" containerID="cri-o://62dacbaae253a9a1de122e5b1ae1ab354fdf5afdf55bc1051348097fb8166208" gracePeriod=30 Oct 30 21:03:03.433665 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:03.433631 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn"] Oct 30 21:03:03.437503 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:03.433882 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn" podUID="4e19f2d2-409b-457e-8f87-d4308b2c53ff" containerName="payload-pre-processing" containerID="cri-o://f70ca371d26e48325480616d746cb16e90bdbd2d9ed6c85c51839609e45b59df" gracePeriod=30 Oct 30 21:03:03.684891 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:03.683586 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:03:03.897826 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:03.784053 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:03:04.208352 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:04.208298 4848 generic.go:356] "Generic (PLEG): container finished" podID="eaaa8712-2a3c-42b8-897d-248d330bd675" containerID="62dacbaae253a9a1de122e5b1ae1ab354fdf5afdf55bc1051348097fb8166208" exitCode=0 Oct 30 21:03:04.211079 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:04.211056 4848 generic.go:356] "Generic (PLEG): container finished" podID="4e19f2d2-409b-457e-8f87-d4308b2c53ff" containerID="f70ca371d26e48325480616d746cb16e90bdbd2d9ed6c85c51839609e45b59df" exitCode=0 Oct 30 21:03:04.212373 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:04.208515 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48" event={"ID":"eaaa8712-2a3c-42b8-897d-248d330bd675","Type":"ContainerDied","Data":"62dacbaae253a9a1de122e5b1ae1ab354fdf5afdf55bc1051348097fb8166208"} Oct 30 21:03:04.212710 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:04.212689 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn" event={"ID":"4e19f2d2-409b-457e-8f87-d4308b2c53ff","Type":"ContainerDied","Data":"f70ca371d26e48325480616d746cb16e90bdbd2d9ed6c85c51839609e45b59df"} Oct 30 21:03:04.238820 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:04.238586 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48" Oct 30 21:03:04.265445 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:04.265416 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn" Oct 30 21:03:04.402595 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:04.397550 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e19f2d2-409b-457e-8f87-d4308b2c53ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e19f2d2-409b-457e-8f87-d4308b2c53ff-plugins-config-volume\") pod \"4e19f2d2-409b-457e-8f87-d4308b2c53ff\" (UID: \"4e19f2d2-409b-457e-8f87-d4308b2c53ff\") " Oct 30 21:03:04.402595 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:04.397617 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaaa8712-2a3c-42b8-897d-248d330bd675-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaaa8712-2a3c-42b8-897d-248d330bd675-plugins-config-volume\") pod \"eaaa8712-2a3c-42b8-897d-248d330bd675\" (UID: \"eaaa8712-2a3c-42b8-897d-248d330bd675\") " Oct 30 21:03:04.402595 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:04.397741 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaaa8712-2a3c-42b8-897d-248d330bd675-kube-api-access-lv58c\" (UniqueName: \"kubernetes.io/projected/eaaa8712-2a3c-42b8-897d-248d330bd675-kube-api-access-lv58c\") pod \"eaaa8712-2a3c-42b8-897d-248d330bd675\" (UID: \"eaaa8712-2a3c-42b8-897d-248d330bd675\") " Oct 30 21:03:04.402595 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:04.397911 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e19f2d2-409b-457e-8f87-d4308b2c53ff-kube-api-access-h9rhc\" (UniqueName: \"kubernetes.io/projected/4e19f2d2-409b-457e-8f87-d4308b2c53ff-kube-api-access-h9rhc\") pod \"4e19f2d2-409b-457e-8f87-d4308b2c53ff\" (UID: \"4e19f2d2-409b-457e-8f87-d4308b2c53ff\") " Oct 30 21:03:04.402595 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:04.401562 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaaa8712-2a3c-42b8-897d-248d330bd675-plugins-config-volume" pod "eaaa8712-2a3c-42b8-897d-248d330bd675" (UID: "eaaa8712-2a3c-42b8-897d-248d330bd675"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:03:04.402595 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:04.401740 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e19f2d2-409b-457e-8f87-d4308b2c53ff-plugins-config-volume" pod "4e19f2d2-409b-457e-8f87-d4308b2c53ff" (UID: "4e19f2d2-409b-457e-8f87-d4308b2c53ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:03:04.406857 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:04.406819 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaaa8712-2a3c-42b8-897d-248d330bd675-kube-api-access-lv58c" pod "eaaa8712-2a3c-42b8-897d-248d330bd675" (UID: "eaaa8712-2a3c-42b8-897d-248d330bd675"). InnerVolumeSpecName "kube-api-access-lv58c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:03:04.415739 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:04.412320 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e19f2d2-409b-457e-8f87-d4308b2c53ff-kube-api-access-h9rhc" pod "4e19f2d2-409b-457e-8f87-d4308b2c53ff" (UID: "4e19f2d2-409b-457e-8f87-d4308b2c53ff"). InnerVolumeSpecName "kube-api-access-h9rhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:03:04.500441 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:04.500357 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e19f2d2-409b-457e-8f87-d4308b2c53ff-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:04.500441 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:04.500390 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaaa8712-2a3c-42b8-897d-248d330bd675-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:04.500441 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:04.500400 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv58c\" (UniqueName: \"kubernetes.io/projected/eaaa8712-2a3c-42b8-897d-248d330bd675-kube-api-access-lv58c\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:04.500441 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:04.500409 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9rhc\" (UniqueName: \"kubernetes.io/projected/4e19f2d2-409b-457e-8f87-d4308b2c53ff-kube-api-access-h9rhc\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:05.220226 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:05.220149 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn" Oct 30 21:03:05.220226 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:05.220181 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn" event={"ID":"4e19f2d2-409b-457e-8f87-d4308b2c53ff","Type":"ContainerDied","Data":"42d41711524a02fea7e938dddc7473548478d4b2285b2c068fd7851adad03cfc"} Oct 30 21:03:05.220726 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:05.220242 4848 scope.go:122] "RemoveContainer" containerID="f70ca371d26e48325480616d746cb16e90bdbd2d9ed6c85c51839609e45b59df" Oct 30 21:03:05.222060 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:05.222038 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48" Oct 30 21:03:05.222179 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:05.222068 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48" event={"ID":"eaaa8712-2a3c-42b8-897d-248d330bd675","Type":"ContainerDied","Data":"97f8208e99c8a841e03f9c0772fb0b34bea887b994692472dfe4fec64b39304c"} Oct 30 21:03:05.230748 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:05.230722 4848 scope.go:122] "RemoveContainer" containerID="62dacbaae253a9a1de122e5b1ae1ab354fdf5afdf55bc1051348097fb8166208" Oct 30 21:03:05.242918 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:05.242891 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn"] Oct 30 21:03:05.246416 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:05.246394 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b46fb7-5577bf5499-cdvwn"] Oct 30 21:03:05.255561 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:05.255539 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48"] Oct 30 21:03:05.263060 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:05.263029 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b46fb7-8d79db7c6-c9s48"] Oct 30 21:03:06.934107 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:06.934074 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e19f2d2-409b-457e-8f87-d4308b2c53ff" path="/var/lib/kubelet/pods/4e19f2d2-409b-457e-8f87-d4308b2c53ff/volumes" Oct 30 21:03:06.934564 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:06.934549 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaaa8712-2a3c-42b8-897d-248d330bd675" path="/var/lib/kubelet/pods/eaaa8712-2a3c-42b8-897d-248d330bd675/volumes" Oct 30 21:03:08.214841 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.214783 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck"] Oct 30 21:03:08.215823 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.215800 4848 cpu_manager.go:414] "containerMap: removing container" podUID="4e19f2d2-409b-457e-8f87-d4308b2c53ff" containerName="payload-pre-processing" Oct 30 21:03:08.215823 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.215822 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e19f2d2-409b-457e-8f87-d4308b2c53ff" containerName="payload-pre-processing" Oct 30 21:03:08.215998 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.215848 4848 cpu_manager.go:414] "containerMap: removing container" podUID="eaaa8712-2a3c-42b8-897d-248d330bd675" containerName="payload-processing" Oct 30 21:03:08.215998 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.215856 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaaa8712-2a3c-42b8-897d-248d330bd675" containerName="payload-processing" Oct 30 21:03:08.215998 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.215875 4848 cpu_manager.go:414] "containerMap: removing container" podUID="5dd5f82e-2c41-4971-ad3b-2bb874ae225c" containerName="payload-pre-processing" Oct 30 21:03:08.215998 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.215883 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd5f82e-2c41-4971-ad3b-2bb874ae225c" containerName="payload-pre-processing" Oct 30 21:03:08.215998 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.215931 4848 cpu_manager.go:414] "containerMap: removing container" podUID="cc8792ce-9e3b-47c0-93a5-e5a7ded65612" containerName="payload-processing" Oct 30 21:03:08.215998 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.215939 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8792ce-9e3b-47c0-93a5-e5a7ded65612" containerName="payload-processing" Oct 30 21:03:08.216208 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.216116 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8792ce-9e3b-47c0-93a5-e5a7ded65612" containerName="payload-processing" Oct 30 21:03:08.216208 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.216135 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e19f2d2-409b-457e-8f87-d4308b2c53ff" containerName="payload-pre-processing" Oct 30 21:03:08.216208 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.216147 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaaa8712-2a3c-42b8-897d-248d330bd675" containerName="payload-processing" Oct 30 21:03:08.216208 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.216155 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd5f82e-2c41-4971-ad3b-2bb874ae225c" containerName="payload-pre-processing" Oct 30 21:03:08.222065 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.222043 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" Oct 30 21:03:08.232000 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.231975 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-f64640-serving-cert\"" Oct 30 21:03:08.233640 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.233615 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck"] Oct 30 21:03:08.282989 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.282962 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72874e47-9bf8-4fcd-98b8-4c51b124244e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck\" (UID: \"72874e47-9bf8-4fcd-98b8-4c51b124244e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" Oct 30 21:03:08.283168 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.283005 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72874e47-9bf8-4fcd-98b8-4c51b124244e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck\" (UID: \"72874e47-9bf8-4fcd-98b8-4c51b124244e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" Oct 30 21:03:08.283168 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.283034 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72874e47-9bf8-4fcd-98b8-4c51b124244e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck\" (UID: \"72874e47-9bf8-4fcd-98b8-4c51b124244e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" Oct 30 21:03:08.283271 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.283198 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79mqc\" (UniqueName: \"kubernetes.io/projected/72874e47-9bf8-4fcd-98b8-4c51b124244e-kube-api-access-79mqc\") pod \"maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck\" (UID: \"72874e47-9bf8-4fcd-98b8-4c51b124244e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" Oct 30 21:03:08.385162 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.385132 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72874e47-9bf8-4fcd-98b8-4c51b124244e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck\" (UID: \"72874e47-9bf8-4fcd-98b8-4c51b124244e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" Oct 30 21:03:08.386421 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.386397 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72874e47-9bf8-4fcd-98b8-4c51b124244e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck\" (UID: \"72874e47-9bf8-4fcd-98b8-4c51b124244e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" Oct 30 21:03:08.387059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.387038 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72874e47-9bf8-4fcd-98b8-4c51b124244e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck\" (UID: \"72874e47-9bf8-4fcd-98b8-4c51b124244e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" Oct 30 21:03:08.387636 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.387616 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79mqc\" (UniqueName: \"kubernetes.io/projected/72874e47-9bf8-4fcd-98b8-4c51b124244e-kube-api-access-79mqc\") pod \"maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck\" (UID: \"72874e47-9bf8-4fcd-98b8-4c51b124244e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" Oct 30 21:03:08.388352 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.386221 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72874e47-9bf8-4fcd-98b8-4c51b124244e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck\" (UID: \"72874e47-9bf8-4fcd-98b8-4c51b124244e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" Oct 30 21:03:08.405460 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.405430 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72874e47-9bf8-4fcd-98b8-4c51b124244e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck\" (UID: \"72874e47-9bf8-4fcd-98b8-4c51b124244e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" Oct 30 21:03:08.412664 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.412629 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72874e47-9bf8-4fcd-98b8-4c51b124244e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck\" (UID: \"72874e47-9bf8-4fcd-98b8-4c51b124244e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" Oct 30 21:03:08.424787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.419298 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79mqc\" (UniqueName: \"kubernetes.io/projected/72874e47-9bf8-4fcd-98b8-4c51b124244e-kube-api-access-79mqc\") pod \"maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck\" (UID: \"72874e47-9bf8-4fcd-98b8-4c51b124244e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" Oct 30 21:03:08.547067 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.546980 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" Oct 30 21:03:08.835506 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.835480 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck"] Oct 30 21:03:08.836269 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:03:08.836243 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72874e47_9bf8_4fcd_98b8_4c51b124244e.slice/crio-086569921a70544de0730eb75466158a13d1f869e464438b614db0d22830c9fb WatchSource:0}: Error finding container 086569921a70544de0730eb75466158a13d1f869e464438b614db0d22830c9fb: Status 404 returned error can't find the container with id 086569921a70544de0730eb75466158a13d1f869e464438b614db0d22830c9fb Oct 30 21:03:08.838287 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:08.838265 4848 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:03:09.341838 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:09.341509 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" event={"ID":"72874e47-9bf8-4fcd-98b8-4c51b124244e","Type":"ContainerStarted","Data":"086569921a70544de0730eb75466158a13d1f869e464438b614db0d22830c9fb"} Oct 30 21:03:09.657867 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:09.657779 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z"] Oct 30 21:03:10.400698 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:10.400649 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" event={"ID":"72874e47-9bf8-4fcd-98b8-4c51b124244e","Type":"ContainerStarted","Data":"48b710eb9b0aa7f5696acb05f9d6a1c124f6cf1ef5d15921a3c5a29b15651c69"} Oct 30 21:03:10.429078 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:10.402976 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" Oct 30 21:03:10.496071 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:10.496007 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" podStartSLOduration=1.9330554979999999 podStartE2EDuration="2.495984841s" totalImagesPullingTime="562.929343ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:10.441212009 +0000 UTC m=+1495.621108512" watchObservedRunningTime="2026-10-30 21:03:10.495984841 +0000 UTC m=+1495.675881349" Oct 30 21:03:14.199090 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:14.199053 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f"] Oct 30 21:03:14.205641 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:14.205606 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" Oct 30 21:03:14.215725 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:14.215691 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f"] Oct 30 21:03:14.216084 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:14.216059 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-321ccd-serving-cert\"" Oct 30 21:03:14.301289 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:14.301245 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8mwb\" (UniqueName: \"kubernetes.io/projected/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-kube-api-access-q8mwb\") pod \"maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f\" (UID: \"01342f11-19ac-41fe-87e2-d07bc8c3c4ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" Oct 30 21:03:14.301497 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:14.301473 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f\" (UID: \"01342f11-19ac-41fe-87e2-d07bc8c3c4ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" Oct 30 21:03:14.301580 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:14.301515 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f\" (UID: \"01342f11-19ac-41fe-87e2-d07bc8c3c4ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" Oct 30 21:03:14.301637 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:14.301594 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f\" (UID: \"01342f11-19ac-41fe-87e2-d07bc8c3c4ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" Oct 30 21:03:14.409793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:14.406978 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f\" (UID: \"01342f11-19ac-41fe-87e2-d07bc8c3c4ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" Oct 30 21:03:14.409793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:14.407042 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f\" (UID: \"01342f11-19ac-41fe-87e2-d07bc8c3c4ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" Oct 30 21:03:14.409793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:14.407078 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f\" (UID: \"01342f11-19ac-41fe-87e2-d07bc8c3c4ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" Oct 30 21:03:14.409793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:14.407309 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8mwb\" (UniqueName: \"kubernetes.io/projected/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-kube-api-access-q8mwb\") pod \"maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f\" (UID: \"01342f11-19ac-41fe-87e2-d07bc8c3c4ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" Oct 30 21:03:14.414453 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:14.414422 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f\" (UID: \"01342f11-19ac-41fe-87e2-d07bc8c3c4ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" Oct 30 21:03:14.461865 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:14.445579 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8mwb\" (UniqueName: \"kubernetes.io/projected/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-kube-api-access-q8mwb\") pod \"maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f\" (UID: \"01342f11-19ac-41fe-87e2-d07bc8c3c4ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" Oct 30 21:03:14.485791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:14.469939 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f\" (UID: \"01342f11-19ac-41fe-87e2-d07bc8c3c4ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" Oct 30 21:03:14.499844 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:14.495708 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f\" (UID: \"01342f11-19ac-41fe-87e2-d07bc8c3c4ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" Oct 30 21:03:14.534251 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:14.534224 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" Oct 30 21:03:14.932148 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:14.929092 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f"] Oct 30 21:03:14.938653 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:03:14.938622 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01342f11_19ac_41fe_87e2_d07bc8c3c4ba.slice/crio-9e4857a620a4d64caf52ce9d57e627c3f005c67cca73f6b53d8fe0b32627fc0a WatchSource:0}: Error finding container 9e4857a620a4d64caf52ce9d57e627c3f005c67cca73f6b53d8fe0b32627fc0a: Status 404 returned error can't find the container with id 9e4857a620a4d64caf52ce9d57e627c3f005c67cca73f6b53d8fe0b32627fc0a Oct 30 21:03:15.522472 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:15.522429 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" event={"ID":"01342f11-19ac-41fe-87e2-d07bc8c3c4ba","Type":"ContainerStarted","Data":"dd525585077086397541ff2dec2009d313646dc9bf8fe2d32da6f2c6681c6f73"} Oct 30 21:03:15.522880 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:15.522492 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" event={"ID":"01342f11-19ac-41fe-87e2-d07bc8c3c4ba","Type":"ContainerStarted","Data":"9e4857a620a4d64caf52ce9d57e627c3f005c67cca73f6b53d8fe0b32627fc0a"} Oct 30 21:03:15.522880 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:15.522527 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" Oct 30 21:03:15.541108 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:15.541048 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" podStartSLOduration=1.0928602729999999 podStartE2EDuration="1.541029111s" totalImagesPullingTime="448.168838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:15.53954876 +0000 UTC m=+1500.719445264" watchObservedRunningTime="2026-10-30 21:03:15.541029111 +0000 UTC m=+1500.720925619" Oct 30 21:03:15.986960 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:15.986928 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 21:03:15.992139 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:15.992117 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 21:03:17.458133 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:17.458101 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" Oct 30 21:03:21.567921 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:21.567887 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" Oct 30 21:03:22.743792 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:03:22.741943 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:03:24.681572 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:24.681499 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" podUID="14ba0389-1004-4f23-8560-cf6fe6f597dd" containerName="main" containerID="cri-o://062aaa1f6d33641289152a8dfab16f311c6fb51dd038cee4ba4d20ec1b6cc91b" gracePeriod=45 Oct 30 21:03:24.961283 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:24.961258 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:03:25.047004 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.046968 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-model-cache\") pod \"14ba0389-1004-4f23-8560-cf6fe6f597dd\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " Oct 30 21:03:25.047196 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.047141 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-dshm\") pod \"14ba0389-1004-4f23-8560-cf6fe6f597dd\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " Oct 30 21:03:25.047258 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.047217 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-model-cache" pod "14ba0389-1004-4f23-8560-cf6fe6f597dd" (UID: "14ba0389-1004-4f23-8560-cf6fe6f597dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:03:25.047311 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.047300 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14ba0389-1004-4f23-8560-cf6fe6f597dd-kube-api-access-qmv2n\" (UniqueName: \"kubernetes.io/projected/14ba0389-1004-4f23-8560-cf6fe6f597dd-kube-api-access-qmv2n\") pod \"14ba0389-1004-4f23-8560-cf6fe6f597dd\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " Oct 30 21:03:25.047377 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.047361 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-tmp-dir\") pod \"14ba0389-1004-4f23-8560-cf6fe6f597dd\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " Oct 30 21:03:25.047457 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.047445 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-home\" (UniqueName: \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-home\") pod \"14ba0389-1004-4f23-8560-cf6fe6f597dd\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " Oct 30 21:03:25.047525 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.047512 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14ba0389-1004-4f23-8560-cf6fe6f597dd-tls-certs\" (UniqueName: \"kubernetes.io/secret/14ba0389-1004-4f23-8560-cf6fe6f597dd-tls-certs\") pod \"14ba0389-1004-4f23-8560-cf6fe6f597dd\" (UID: \"14ba0389-1004-4f23-8560-cf6fe6f597dd\") " Oct 30 21:03:25.047626 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.047594 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-tmp-dir" pod "14ba0389-1004-4f23-8560-cf6fe6f597dd" (UID: "14ba0389-1004-4f23-8560-cf6fe6f597dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:03:25.047779 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.047740 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-home" pod "14ba0389-1004-4f23-8560-cf6fe6f597dd" (UID: "14ba0389-1004-4f23-8560-cf6fe6f597dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:03:25.048073 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.048056 4848 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-tmp-dir\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:25.048157 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.048077 4848 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-home\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:25.048157 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.048090 4848 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-model-cache\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:25.049469 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.049443 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-dshm" pod "14ba0389-1004-4f23-8560-cf6fe6f597dd" (UID: "14ba0389-1004-4f23-8560-cf6fe6f597dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:03:25.049556 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.049516 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14ba0389-1004-4f23-8560-cf6fe6f597dd-tls-certs" pod "14ba0389-1004-4f23-8560-cf6fe6f597dd" (UID: "14ba0389-1004-4f23-8560-cf6fe6f597dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:03:25.049745 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.049730 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ba0389-1004-4f23-8560-cf6fe6f597dd-kube-api-access-qmv2n" pod "14ba0389-1004-4f23-8560-cf6fe6f597dd" (UID: "14ba0389-1004-4f23-8560-cf6fe6f597dd"). InnerVolumeSpecName "kube-api-access-qmv2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:03:25.149843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.149813 4848 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14ba0389-1004-4f23-8560-cf6fe6f597dd-tls-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:25.149843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.149840 4848 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14ba0389-1004-4f23-8560-cf6fe6f597dd-dshm\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:25.149843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.149850 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmv2n\" (UniqueName: \"kubernetes.io/projected/14ba0389-1004-4f23-8560-cf6fe6f597dd-kube-api-access-qmv2n\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:25.719970 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.719937 4848 generic.go:356] "Generic (PLEG): container finished" podID="14ba0389-1004-4f23-8560-cf6fe6f597dd" containerID="062aaa1f6d33641289152a8dfab16f311c6fb51dd038cee4ba4d20ec1b6cc91b" exitCode=0 Oct 30 21:03:25.720389 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.719983 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" Oct 30 21:03:25.720389 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.720019 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" event={"ID":"14ba0389-1004-4f23-8560-cf6fe6f597dd","Type":"ContainerDied","Data":"062aaa1f6d33641289152a8dfab16f311c6fb51dd038cee4ba4d20ec1b6cc91b"} Oct 30 21:03:25.720389 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.720066 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z" event={"ID":"14ba0389-1004-4f23-8560-cf6fe6f597dd","Type":"ContainerDied","Data":"381c85b9c4487f139b49dd4fe90a93aaf85cb565b86f92e485ae9ed4747b752c"} Oct 30 21:03:25.720389 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.720082 4848 scope.go:122] "RemoveContainer" containerID="062aaa1f6d33641289152a8dfab16f311c6fb51dd038cee4ba4d20ec1b6cc91b" Oct 30 21:03:25.730900 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.730881 4848 scope.go:122] "RemoveContainer" containerID="062aaa1f6d33641289152a8dfab16f311c6fb51dd038cee4ba4d20ec1b6cc91b" Oct 30 21:03:25.731163 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:03:25.731148 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"062aaa1f6d33641289152a8dfab16f311c6fb51dd038cee4ba4d20ec1b6cc91b\": container with ID starting with 062aaa1f6d33641289152a8dfab16f311c6fb51dd038cee4ba4d20ec1b6cc91b not found: ID does not exist" containerID="062aaa1f6d33641289152a8dfab16f311c6fb51dd038cee4ba4d20ec1b6cc91b" Oct 30 21:03:25.731205 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.731171 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"062aaa1f6d33641289152a8dfab16f311c6fb51dd038cee4ba4d20ec1b6cc91b"} err="failed to get container status \"062aaa1f6d33641289152a8dfab16f311c6fb51dd038cee4ba4d20ec1b6cc91b\": rpc error: code = NotFound desc = could not find container \"062aaa1f6d33641289152a8dfab16f311c6fb51dd038cee4ba4d20ec1b6cc91b\": container with ID starting with 062aaa1f6d33641289152a8dfab16f311c6fb51dd038cee4ba4d20ec1b6cc91b not found: ID does not exist" Oct 30 21:03:25.746510 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.746490 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z"] Oct 30 21:03:25.750920 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:25.750897 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-b46fb7/ipp-route-b46fb7-kserve-6b4fd559d9-j6g5z"] Oct 30 21:03:26.935735 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:26.935704 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14ba0389-1004-4f23-8560-cf6fe6f597dd" path="/var/lib/kubelet/pods/14ba0389-1004-4f23-8560-cf6fe6f597dd/volumes" Oct 30 21:03:31.165115 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:31.165081 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg"] Oct 30 21:03:31.165814 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:31.165787 4848 cpu_manager.go:414] "containerMap: removing container" podUID="14ba0389-1004-4f23-8560-cf6fe6f597dd" containerName="main" Oct 30 21:03:31.165814 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:31.165807 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ba0389-1004-4f23-8560-cf6fe6f597dd" containerName="main" Oct 30 21:03:31.165928 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:31.165919 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="14ba0389-1004-4f23-8560-cf6fe6f597dd" containerName="main" Oct 30 21:03:31.186264 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:31.184537 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg"] Oct 30 21:03:31.186264 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:31.184654 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" Oct 30 21:03:31.191066 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:31.191040 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-3a65ec-serving-cert\"" Oct 30 21:03:31.288150 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:31.288116 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26q7r\" (UniqueName: \"kubernetes.io/projected/3974a64c-bb4d-436e-bcfe-da5277f71f8c-kube-api-access-26q7r\") pod \"maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg\" (UID: \"3974a64c-bb4d-436e-bcfe-da5277f71f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" Oct 30 21:03:31.288328 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:31.288178 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3974a64c-bb4d-436e-bcfe-da5277f71f8c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg\" (UID: \"3974a64c-bb4d-436e-bcfe-da5277f71f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" Oct 30 21:03:31.288383 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:31.288343 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3974a64c-bb4d-436e-bcfe-da5277f71f8c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg\" (UID: \"3974a64c-bb4d-436e-bcfe-da5277f71f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" Oct 30 21:03:31.288459 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:31.288441 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3974a64c-bb4d-436e-bcfe-da5277f71f8c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg\" (UID: \"3974a64c-bb4d-436e-bcfe-da5277f71f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" Oct 30 21:03:31.390395 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:31.390361 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3974a64c-bb4d-436e-bcfe-da5277f71f8c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg\" (UID: \"3974a64c-bb4d-436e-bcfe-da5277f71f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" Oct 30 21:03:31.390581 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:31.390564 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26q7r\" (UniqueName: \"kubernetes.io/projected/3974a64c-bb4d-436e-bcfe-da5277f71f8c-kube-api-access-26q7r\") pod \"maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg\" (UID: \"3974a64c-bb4d-436e-bcfe-da5277f71f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" Oct 30 21:03:31.390666 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:31.390625 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3974a64c-bb4d-436e-bcfe-da5277f71f8c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg\" (UID: \"3974a64c-bb4d-436e-bcfe-da5277f71f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" Oct 30 21:03:31.390755 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:31.390740 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3974a64c-bb4d-436e-bcfe-da5277f71f8c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg\" (UID: \"3974a64c-bb4d-436e-bcfe-da5277f71f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" Oct 30 21:03:31.391212 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:31.391182 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3974a64c-bb4d-436e-bcfe-da5277f71f8c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg\" (UID: \"3974a64c-bb4d-436e-bcfe-da5277f71f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" Oct 30 21:03:31.394290 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:31.394266 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3974a64c-bb4d-436e-bcfe-da5277f71f8c-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg\" (UID: \"3974a64c-bb4d-436e-bcfe-da5277f71f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" Oct 30 21:03:31.400655 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:31.400628 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3974a64c-bb4d-436e-bcfe-da5277f71f8c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg\" (UID: \"3974a64c-bb4d-436e-bcfe-da5277f71f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" Oct 30 21:03:31.411986 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:31.411915 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26q7r\" (UniqueName: \"kubernetes.io/projected/3974a64c-bb4d-436e-bcfe-da5277f71f8c-kube-api-access-26q7r\") pod \"maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg\" (UID: \"3974a64c-bb4d-436e-bcfe-da5277f71f8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" Oct 30 21:03:31.505443 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:31.505405 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" Oct 30 21:03:32.101875 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:32.101843 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg"] Oct 30 21:03:32.810733 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:32.810632 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" event={"ID":"3974a64c-bb4d-436e-bcfe-da5277f71f8c","Type":"ContainerStarted","Data":"4bf13d21f81a2bfc1226d7f0ef0d022fb5ddd4233e63a418035e6e15769be911"} Oct 30 21:03:32.810733 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:32.810703 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" event={"ID":"3974a64c-bb4d-436e-bcfe-da5277f71f8c","Type":"ContainerStarted","Data":"e8f3ed87dbca4ce26f8783607272ef8a9c7f22da2e43d70ff0cfe8da3458bfc6"} Oct 30 21:03:32.810733 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:32.810732 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" Oct 30 21:03:32.845171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:32.845111 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" podStartSLOduration=1.385204122 podStartE2EDuration="1.845096268s" totalImagesPullingTime="459.892146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:32.840326884 +0000 UTC m=+1518.020223382" watchObservedRunningTime="2026-10-30 21:03:32.845096268 +0000 UTC m=+1518.024992775" Oct 30 21:03:38.658434 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:38.658404 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc"] Oct 30 21:03:38.658984 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:38.658952 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" podUID="0226107b-80ef-4c6e-888c-f89f73108c20" containerName="istio-proxy" containerID="cri-o://072192f1b8a41c1fa6a026efc64d597a75285d036e5aac6af8f32ac70860f210" gracePeriod=30 Oct 30 21:03:38.897550 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:38.897516 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" Oct 30 21:03:43.997937 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:43.997900 4848 generic.go:356] "Generic (PLEG): container finished" podID="0226107b-80ef-4c6e-888c-f89f73108c20" containerID="072192f1b8a41c1fa6a026efc64d597a75285d036e5aac6af8f32ac70860f210" exitCode=0 Oct 30 21:03:43.998379 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:43.997984 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" event={"ID":"0226107b-80ef-4c6e-888c-f89f73108c20","Type":"ContainerDied","Data":"072192f1b8a41c1fa6a026efc64d597a75285d036e5aac6af8f32ac70860f210"} Oct 30 21:03:43.998379 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:43.998065 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" event={"ID":"0226107b-80ef-4c6e-888c-f89f73108c20","Type":"ContainerDied","Data":"c7f8a1b05245e229656320ea7e7bb2633f0f8d6cfde4b68092a2d5c76d2dd64b"} Oct 30 21:03:43.998379 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:43.998092 4848 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7f8a1b05245e229656320ea7e7bb2633f0f8d6cfde4b68092a2d5c76d2dd64b" Oct 30 21:03:44.042501 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.040505 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.144625 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-istio-envoy\") pod \"0226107b-80ef-4c6e-888c-f89f73108c20\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.144718 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-workload-certs\") pod \"0226107b-80ef-4c6e-888c-f89f73108c20\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.145051 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0226107b-80ef-4c6e-888c-f89f73108c20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0226107b-80ef-4c6e-888c-f89f73108c20-istiod-ca-cert\") pod \"0226107b-80ef-4c6e-888c-f89f73108c20\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.145154 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0226107b-80ef-4c6e-888c-f89f73108c20-kube-api-access-bvrkf\" (UniqueName: \"kubernetes.io/projected/0226107b-80ef-4c6e-888c-f89f73108c20-kube-api-access-bvrkf\") pod \"0226107b-80ef-4c6e-888c-f89f73108c20\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.145172 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-workload-certs" pod "0226107b-80ef-4c6e-888c-f89f73108c20" (UID: "0226107b-80ef-4c6e-888c-f89f73108c20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.145206 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-istio-data\") pod \"0226107b-80ef-4c6e-888c-f89f73108c20\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.145447 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-credential-socket\") pod \"0226107b-80ef-4c6e-888c-f89f73108c20\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.145453 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-istio-data" pod "0226107b-80ef-4c6e-888c-f89f73108c20" (UID: "0226107b-80ef-4c6e-888c-f89f73108c20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.145487 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0226107b-80ef-4c6e-888c-f89f73108c20-istio-token\" (UniqueName: \"kubernetes.io/projected/0226107b-80ef-4c6e-888c-f89f73108c20-istio-token\") pod \"0226107b-80ef-4c6e-888c-f89f73108c20\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.145566 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-workload-socket\") pod \"0226107b-80ef-4c6e-888c-f89f73108c20\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.145613 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0226107b-80ef-4c6e-888c-f89f73108c20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0226107b-80ef-4c6e-888c-f89f73108c20-istio-podinfo\") pod \"0226107b-80ef-4c6e-888c-f89f73108c20\" (UID: \"0226107b-80ef-4c6e-888c-f89f73108c20\") " Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.145749 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0226107b-80ef-4c6e-888c-f89f73108c20-istiod-ca-cert" pod "0226107b-80ef-4c6e-888c-f89f73108c20" (UID: "0226107b-80ef-4c6e-888c-f89f73108c20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.146341 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.146365 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0226107b-80ef-4c6e-888c-f89f73108c20-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.146380 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.147359 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-credential-socket" pod "0226107b-80ef-4c6e-888c-f89f73108c20" (UID: "0226107b-80ef-4c6e-888c-f89f73108c20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.147423 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-workload-socket" pod "0226107b-80ef-4c6e-888c-f89f73108c20" (UID: "0226107b-80ef-4c6e-888c-f89f73108c20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.150721 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0226107b-80ef-4c6e-888c-f89f73108c20-kube-api-access-bvrkf" pod "0226107b-80ef-4c6e-888c-f89f73108c20" (UID: "0226107b-80ef-4c6e-888c-f89f73108c20"). InnerVolumeSpecName "kube-api-access-bvrkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.150956 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-istio-envoy" pod "0226107b-80ef-4c6e-888c-f89f73108c20" (UID: "0226107b-80ef-4c6e-888c-f89f73108c20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.151703 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0226107b-80ef-4c6e-888c-f89f73108c20-istio-token" pod "0226107b-80ef-4c6e-888c-f89f73108c20" (UID: "0226107b-80ef-4c6e-888c-f89f73108c20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:03:44.161410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.153462 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0226107b-80ef-4c6e-888c-f89f73108c20-istio-podinfo" pod "0226107b-80ef-4c6e-888c-f89f73108c20" (UID: "0226107b-80ef-4c6e-888c-f89f73108c20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:03:44.609721 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.247497 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:44.609721 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.247522 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0226107b-80ef-4c6e-888c-f89f73108c20-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:44.609721 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.247535 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:44.609721 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.247543 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0226107b-80ef-4c6e-888c-f89f73108c20-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:44.609721 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.247554 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0226107b-80ef-4c6e-888c-f89f73108c20-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:44.609721 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.247564 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvrkf\" (UniqueName: \"kubernetes.io/projected/0226107b-80ef-4c6e-888c-f89f73108c20-kube-api-access-bvrkf\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:03:45.008344 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.963029 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 21:03:45.008344 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:44.963080 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" podUID="01a47632-b985-464f-8ee1-4fe60a1b3258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 21:03:45.008344 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:45.006938 4848 generic.go:356] "Generic (PLEG): container finished" podID="01a47632-b985-464f-8ee1-4fe60a1b3258" containerID="6adfe6528c1330c09fabb1d8b4ab1f784850138591c7147430f5c3f18fba47f5" exitCode=0 Oct 30 21:03:45.008344 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:45.007018 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" event={"ID":"01a47632-b985-464f-8ee1-4fe60a1b3258","Type":"ContainerDied","Data":"6adfe6528c1330c09fabb1d8b4ab1f784850138591c7147430f5c3f18fba47f5"} Oct 30 21:03:45.008344 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:45.007187 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc" Oct 30 21:03:45.008344 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:45.007755 4848 scope.go:122] "RemoveContainer" containerID="6adfe6528c1330c09fabb1d8b4ab1f784850138591c7147430f5c3f18fba47f5" Oct 30 21:03:45.009738 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:45.009699 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:03:45.009893 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:45.009832 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:03:45.009893 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:45.009884 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:03:45.029038 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:45.029018 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc"] Oct 30 21:03:45.037386 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:45.037359 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-mfvmc"] Oct 30 21:03:46.040203 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:46.040169 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" event={"ID":"01a47632-b985-464f-8ee1-4fe60a1b3258","Type":"ContainerStarted","Data":"9961db2fc9b44abf0a714d9150b821813502338cedeeefa564ef085cdb3ad483"} Oct 30 21:03:46.042748 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:46.042711 4848 generic.go:356] "Generic (PLEG): container finished" podID="7f1686ee-2109-4bfa-9964-cc40955a35b9" containerID="794183a3dee72e74d092a786187b0bc7b8acf59556b005b53f7359353a405e26" exitCode=0 Oct 30 21:03:46.042910 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:46.042822 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" event={"ID":"7f1686ee-2109-4bfa-9964-cc40955a35b9","Type":"ContainerDied","Data":"794183a3dee72e74d092a786187b0bc7b8acf59556b005b53f7359353a405e26"} Oct 30 21:03:46.043314 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:46.043297 4848 scope.go:122] "RemoveContainer" containerID="794183a3dee72e74d092a786187b0bc7b8acf59556b005b53f7359353a405e26" Oct 30 21:03:46.045399 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:46.045367 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:03:46.045506 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:46.045454 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:03:46.045506 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:46.045498 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:03:46.707790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:46.707166 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:03:46.707790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:46.707380 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 21:03:46.707790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:46.707444 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" podUID="01a47632-b985-464f-8ee1-4fe60a1b3258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 21:03:47.141667 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:47.141515 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0226107b-80ef-4c6e-888c-f89f73108c20" path="/var/lib/kubelet/pods/0226107b-80ef-4c6e-888c-f89f73108c20/volumes" Oct 30 21:03:47.144698 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:47.144636 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" event={"ID":"7f1686ee-2109-4bfa-9964-cc40955a35b9","Type":"ContainerStarted","Data":"608b801315fffc5cd5f7ee45d8912e02add5a0fa8241292356be24351cb215d4"} Oct 30 21:03:47.332795 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:47.332034 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:03:47.332795 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:47.332167 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 21:03:47.332795 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:47.332225 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" podUID="7f1686ee-2109-4bfa-9964-cc40955a35b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 21:03:47.723788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:47.717261 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 21:03:47.723788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:47.717388 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" podUID="01a47632-b985-464f-8ee1-4fe60a1b3258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 21:03:48.337364 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:48.337325 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 21:03:48.338157 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:48.338124 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" podUID="7f1686ee-2109-4bfa-9964-cc40955a35b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 21:03:48.713287 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:48.713251 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 21:03:48.713605 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:48.713563 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" podUID="01a47632-b985-464f-8ee1-4fe60a1b3258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 21:03:49.350793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:49.343909 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 21:03:49.350793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:49.343976 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" podUID="7f1686ee-2109-4bfa-9964-cc40955a35b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 21:03:49.716230 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:49.716191 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 21:03:49.716539 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:49.716504 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" podUID="01a47632-b985-464f-8ee1-4fe60a1b3258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 21:03:50.347219 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:50.347179 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 21:03:50.347569 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:50.347540 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" podUID="7f1686ee-2109-4bfa-9964-cc40955a35b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 21:03:50.724790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:50.721222 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 21:03:50.724790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:50.721290 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" podUID="01a47632-b985-464f-8ee1-4fe60a1b3258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 21:03:51.069867 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.069563 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b"] Oct 30 21:03:51.071410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.070797 4848 cpu_manager.go:414] "containerMap: removing container" podUID="0226107b-80ef-4c6e-888c-f89f73108c20" containerName="istio-proxy" Oct 30 21:03:51.071410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.070821 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0226107b-80ef-4c6e-888c-f89f73108c20" containerName="istio-proxy" Oct 30 21:03:51.071410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.071010 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="0226107b-80ef-4c6e-888c-f89f73108c20" containerName="istio-proxy" Oct 30 21:03:51.076541 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.076522 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.080642 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.080576 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b"] Oct 30 21:03:51.082358 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.082320 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-a50675-openshift-default-dockercfg-4pqn7\"" Oct 30 21:03:51.154509 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.154463 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-envoy\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.154688 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.154519 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r68kd\" (UniqueName: \"kubernetes.io/projected/ebde61df-6d3a-43f2-8ab0-e61c220a5259-kube-api-access-r68kd\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.154688 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.154589 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-workload-certs\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.154688 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.154630 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-data\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.154688 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.154664 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istiod-ca-cert\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.155041 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.154790 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-token\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.155041 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.154848 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-workload-socket\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.155041 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.154895 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-podinfo\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.155041 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.154940 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-credential-socket\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.329858 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.312746 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-data\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.388239 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.388203 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istiod-ca-cert\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.396803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.388643 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-token\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.401678 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.397163 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-workload-socket\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.418939 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.418906 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-podinfo\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.430137 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.430100 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-credential-socket\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.461713 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.461674 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-envoy\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.486792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.465848 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r68kd\" (UniqueName: \"kubernetes.io/projected/ebde61df-6d3a-43f2-8ab0-e61c220a5259-kube-api-access-r68kd\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.488031 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.487992 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-workload-certs\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.508896 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.330511 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-data\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.508896 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.363203 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 21:03:51.508896 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.491038 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" podUID="7f1686ee-2109-4bfa-9964-cc40955a35b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 21:03:51.508896 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.392302 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istiod-ca-cert\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.508896 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.418206 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-workload-socket\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.508896 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.441034 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-podinfo\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.509391 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.434380 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-credential-socket\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.510720 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.510679 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-token\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.511464 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.511443 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-workload-certs\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.544848 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.543823 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-envoy\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.564569 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.564532 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r68kd\" (UniqueName: \"kubernetes.io/projected/ebde61df-6d3a-43f2-8ab0-e61c220a5259-kube-api-access-r68kd\") pod \"e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:51.725790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.708181 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 21:03:51.725790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.708246 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" podUID="01a47632-b985-464f-8ee1-4fe60a1b3258" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 21:03:51.757789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:51.756857 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:52.162147 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:03:52.162110 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebde61df_6d3a_43f2_8ab0_e61c220a5259.slice/crio-cd68a58a2d2dbe715ec128423843ff35b0d31039de30cacc15d85c253d039eaa WatchSource:0}: Error finding container cd68a58a2d2dbe715ec128423843ff35b0d31039de30cacc15d85c253d039eaa: Status 404 returned error can't find the container with id cd68a58a2d2dbe715ec128423843ff35b0d31039de30cacc15d85c253d039eaa Oct 30 21:03:52.165951 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:52.165907 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:03:52.166076 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:52.166005 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:03:52.166076 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:52.166051 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:03:52.166306 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:52.166286 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b"] Oct 30 21:03:52.331632 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:52.331179 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 21:03:52.331632 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:52.331239 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" podUID="7f1686ee-2109-4bfa-9964-cc40955a35b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 21:03:52.552107 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:52.552068 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" event={"ID":"ebde61df-6d3a-43f2-8ab0-e61c220a5259","Type":"ContainerStarted","Data":"b5a6137eac4ae7e616ed1b7897b1d7a9b81bad785ab7e96e43ee25468eeaf6bf"} Oct 30 21:03:52.552407 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:52.552382 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" event={"ID":"ebde61df-6d3a-43f2-8ab0-e61c220a5259","Type":"ContainerStarted","Data":"cd68a58a2d2dbe715ec128423843ff35b0d31039de30cacc15d85c253d039eaa"} Oct 30 21:03:52.554014 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:03:52.553985 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:03:52.762809 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:52.758869 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:52.954333 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:52.954305 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:03:52.956082 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:52.956017 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:03:52.957590 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:52.957501 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:03:52.979811 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:52.978417 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" podStartSLOduration=1.978397409 podStartE2EDuration="1.978397409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:03:52.575096326 +0000 UTC m=+1537.754992833" watchObservedRunningTime="2026-10-30 21:03:52.978397409 +0000 UTC m=+1538.158293924" Oct 30 21:03:53.343828 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:53.339194 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 21:03:53.343828 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:53.339258 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" podUID="7f1686ee-2109-4bfa-9964-cc40955a35b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 21:03:53.764209 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:53.764175 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:54.335393 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:54.335366 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:03:54.335673 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:54.335657 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:03:54.336465 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:54.336448 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:03:54.566255 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:54.566222 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:54.567168 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:54.567147 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:03:56.714661 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:56.714627 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr"] Oct 30 21:03:56.715157 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:03:56.714889 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" podUID="5c00d1f3-b911-40fd-8772-3ed4bba10d75" containerName="istio-proxy" containerID="cri-o://e9d93d1afae7368d5a19cd9de7b9d67739392239a1ec0ab41647d36196a89602" gracePeriod=30 Oct 30 21:04:02.077829 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.077799 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 21:04:02.151557 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.151523 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-workload-certs\") pod \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " Oct 30 21:04:02.151756 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.151625 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-token\" (UniqueName: \"kubernetes.io/projected/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-token\") pod \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " Oct 30 21:04:02.151756 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.151705 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-envoy\") pod \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " Oct 30 21:04:02.151943 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.151814 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istiod-ca-cert\") pod \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " Oct 30 21:04:02.151943 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.151869 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-podinfo\") pod \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " Oct 30 21:04:02.151943 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.151886 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-workload-certs" pod "5c00d1f3-b911-40fd-8772-3ed4bba10d75" (UID: "5c00d1f3-b911-40fd-8772-3ed4bba10d75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:02.152110 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.152007 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-data\") pod \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " Oct 30 21:04:02.152110 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.152077 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c00d1f3-b911-40fd-8772-3ed4bba10d75-kube-api-access-dmn42\" (UniqueName: \"kubernetes.io/projected/5c00d1f3-b911-40fd-8772-3ed4bba10d75-kube-api-access-dmn42\") pod \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " Oct 30 21:04:02.152224 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.152152 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-workload-socket\") pod \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " Oct 30 21:04:02.152224 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.152180 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-credential-socket\") pod \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\" (UID: \"5c00d1f3-b911-40fd-8772-3ed4bba10d75\") " Oct 30 21:04:02.153164 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.152814 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:02.153164 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.152957 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istiod-ca-cert" pod "5c00d1f3-b911-40fd-8772-3ed4bba10d75" (UID: "5c00d1f3-b911-40fd-8772-3ed4bba10d75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:04:02.153348 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.153252 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-data" pod "5c00d1f3-b911-40fd-8772-3ed4bba10d75" (UID: "5c00d1f3-b911-40fd-8772-3ed4bba10d75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:02.153504 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.153468 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-workload-socket" pod "5c00d1f3-b911-40fd-8772-3ed4bba10d75" (UID: "5c00d1f3-b911-40fd-8772-3ed4bba10d75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:02.153688 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.153659 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-credential-socket" pod "5c00d1f3-b911-40fd-8772-3ed4bba10d75" (UID: "5c00d1f3-b911-40fd-8772-3ed4bba10d75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:02.154500 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.154469 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-envoy" pod "5c00d1f3-b911-40fd-8772-3ed4bba10d75" (UID: "5c00d1f3-b911-40fd-8772-3ed4bba10d75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:02.154704 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.154684 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c00d1f3-b911-40fd-8772-3ed4bba10d75-kube-api-access-dmn42" pod "5c00d1f3-b911-40fd-8772-3ed4bba10d75" (UID: "5c00d1f3-b911-40fd-8772-3ed4bba10d75"). InnerVolumeSpecName "kube-api-access-dmn42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:02.155510 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.155483 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-podinfo" pod "5c00d1f3-b911-40fd-8772-3ed4bba10d75" (UID: "5c00d1f3-b911-40fd-8772-3ed4bba10d75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:04:02.156370 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.156341 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-token" pod "5c00d1f3-b911-40fd-8772-3ed4bba10d75" (UID: "5c00d1f3-b911-40fd-8772-3ed4bba10d75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:02.254151 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.254101 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmn42\" (UniqueName: \"kubernetes.io/projected/5c00d1f3-b911-40fd-8772-3ed4bba10d75-kube-api-access-dmn42\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:02.254151 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.254142 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:02.254151 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.254157 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:02.254444 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.254171 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:02.254444 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.254183 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:02.254444 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.254198 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:02.254444 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.254212 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:02.254444 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.254224 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c00d1f3-b911-40fd-8772-3ed4bba10d75-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:02.736333 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.736286 4848 generic.go:356] "Generic (PLEG): container finished" podID="5c00d1f3-b911-40fd-8772-3ed4bba10d75" containerID="e9d93d1afae7368d5a19cd9de7b9d67739392239a1ec0ab41647d36196a89602" exitCode=0 Oct 30 21:04:02.736833 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.736753 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" event={"ID":"5c00d1f3-b911-40fd-8772-3ed4bba10d75","Type":"ContainerDied","Data":"e9d93d1afae7368d5a19cd9de7b9d67739392239a1ec0ab41647d36196a89602"} Oct 30 21:04:02.736998 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.736864 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" event={"ID":"5c00d1f3-b911-40fd-8772-3ed4bba10d75","Type":"ContainerDied","Data":"7bdc3cb0854021e5f8e7e0d94ee843dc1f0dc2f1c068e2ffcc367de6767d0360"} Oct 30 21:04:02.736998 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.736896 4848 scope.go:122] "RemoveContainer" containerID="e9d93d1afae7368d5a19cd9de7b9d67739392239a1ec0ab41647d36196a89602" Oct 30 21:04:02.737182 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.737079 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr" Oct 30 21:04:02.775150 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.775126 4848 scope.go:122] "RemoveContainer" containerID="26f56a772ac1cb31314790b81e470d8800e6e09c9790f2c14135946a6b168088" Oct 30 21:04:02.787332 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.787302 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr"] Oct 30 21:04:02.787576 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.787559 4848 scope.go:122] "RemoveContainer" containerID="e9d93d1afae7368d5a19cd9de7b9d67739392239a1ec0ab41647d36196a89602" Oct 30 21:04:02.788290 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:04:02.788265 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9d93d1afae7368d5a19cd9de7b9d67739392239a1ec0ab41647d36196a89602\": container with ID starting with e9d93d1afae7368d5a19cd9de7b9d67739392239a1ec0ab41647d36196a89602 not found: ID does not exist" containerID="e9d93d1afae7368d5a19cd9de7b9d67739392239a1ec0ab41647d36196a89602" Oct 30 21:04:02.788395 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.788296 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9d93d1afae7368d5a19cd9de7b9d67739392239a1ec0ab41647d36196a89602"} err="failed to get container status \"e9d93d1afae7368d5a19cd9de7b9d67739392239a1ec0ab41647d36196a89602\": rpc error: code = NotFound desc = could not find container \"e9d93d1afae7368d5a19cd9de7b9d67739392239a1ec0ab41647d36196a89602\": container with ID starting with e9d93d1afae7368d5a19cd9de7b9d67739392239a1ec0ab41647d36196a89602 not found: ID does not exist" Oct 30 21:04:02.788395 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.788315 4848 scope.go:122] "RemoveContainer" containerID="26f56a772ac1cb31314790b81e470d8800e6e09c9790f2c14135946a6b168088" Oct 30 21:04:02.788625 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:04:02.788605 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f56a772ac1cb31314790b81e470d8800e6e09c9790f2c14135946a6b168088\": container with ID starting with 26f56a772ac1cb31314790b81e470d8800e6e09c9790f2c14135946a6b168088 not found: ID does not exist" containerID="26f56a772ac1cb31314790b81e470d8800e6e09c9790f2c14135946a6b168088" Oct 30 21:04:02.788680 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.788629 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f56a772ac1cb31314790b81e470d8800e6e09c9790f2c14135946a6b168088"} err="failed to get container status \"26f56a772ac1cb31314790b81e470d8800e6e09c9790f2c14135946a6b168088\": rpc error: code = NotFound desc = could not find container \"26f56a772ac1cb31314790b81e470d8800e6e09c9790f2c14135946a6b168088\": container with ID starting with 26f56a772ac1cb31314790b81e470d8800e6e09c9790f2c14135946a6b168088 not found: ID does not exist" Oct 30 21:04:02.791172 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.791152 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b46fb7-openshift-default-99ccd7957-s4xnr"] Oct 30 21:04:02.933729 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:02.933697 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c00d1f3-b911-40fd-8772-3ed4bba10d75" path="/var/lib/kubelet/pods/5c00d1f3-b911-40fd-8772-3ed4bba10d75/volumes" Oct 30 21:04:06.785147 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:06.785110 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq"] Oct 30 21:04:06.786252 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:06.786223 4848 cpu_manager.go:414] "containerMap: removing container" podUID="5c00d1f3-b911-40fd-8772-3ed4bba10d75" containerName="istio-proxy" Oct 30 21:04:06.786431 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:06.786410 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c00d1f3-b911-40fd-8772-3ed4bba10d75" containerName="istio-proxy" Oct 30 21:04:06.786557 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:06.786437 4848 cpu_manager.go:414] "containerMap: removing container" podUID="5c00d1f3-b911-40fd-8772-3ed4bba10d75" containerName="istio-proxy" Oct 30 21:04:06.786557 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:06.786446 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c00d1f3-b911-40fd-8772-3ed4bba10d75" containerName="istio-proxy" Oct 30 21:04:06.786729 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:06.786625 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c00d1f3-b911-40fd-8772-3ed4bba10d75" containerName="istio-proxy" Oct 30 21:04:06.786729 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:06.786641 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c00d1f3-b911-40fd-8772-3ed4bba10d75" containerName="istio-proxy" Oct 30 21:04:06.792274 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:06.792254 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" Oct 30 21:04:06.801103 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:06.801074 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-a50675-serving-cert\"" Oct 30 21:04:06.806601 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:06.806570 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq"] Oct 30 21:04:06.950970 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:06.950742 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/797a8c75-fb91-4d33-8833-e062f5cd01f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq\" (UID: \"797a8c75-fb91-4d33-8833-e062f5cd01f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" Oct 30 21:04:06.951424 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:06.951401 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m62ft\" (UniqueName: \"kubernetes.io/projected/797a8c75-fb91-4d33-8833-e062f5cd01f0-kube-api-access-m62ft\") pod \"maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq\" (UID: \"797a8c75-fb91-4d33-8833-e062f5cd01f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" Oct 30 21:04:06.953154 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:06.951893 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/797a8c75-fb91-4d33-8833-e062f5cd01f0-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq\" (UID: \"797a8c75-fb91-4d33-8833-e062f5cd01f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" Oct 30 21:04:06.953154 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:06.952970 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/797a8c75-fb91-4d33-8833-e062f5cd01f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq\" (UID: \"797a8c75-fb91-4d33-8833-e062f5cd01f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" Oct 30 21:04:07.053983 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:07.053889 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m62ft\" (UniqueName: \"kubernetes.io/projected/797a8c75-fb91-4d33-8833-e062f5cd01f0-kube-api-access-m62ft\") pod \"maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq\" (UID: \"797a8c75-fb91-4d33-8833-e062f5cd01f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" Oct 30 21:04:07.054169 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:07.054012 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/797a8c75-fb91-4d33-8833-e062f5cd01f0-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq\" (UID: \"797a8c75-fb91-4d33-8833-e062f5cd01f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" Oct 30 21:04:07.054169 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:07.054061 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/797a8c75-fb91-4d33-8833-e062f5cd01f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq\" (UID: \"797a8c75-fb91-4d33-8833-e062f5cd01f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" Oct 30 21:04:07.054169 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:07.054111 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/797a8c75-fb91-4d33-8833-e062f5cd01f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq\" (UID: \"797a8c75-fb91-4d33-8833-e062f5cd01f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" Oct 30 21:04:07.054828 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:07.054796 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/797a8c75-fb91-4d33-8833-e062f5cd01f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq\" (UID: \"797a8c75-fb91-4d33-8833-e062f5cd01f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" Oct 30 21:04:07.056590 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:07.056569 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/797a8c75-fb91-4d33-8833-e062f5cd01f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq\" (UID: \"797a8c75-fb91-4d33-8833-e062f5cd01f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" Oct 30 21:04:07.056590 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:07.056584 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/797a8c75-fb91-4d33-8833-e062f5cd01f0-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq\" (UID: \"797a8c75-fb91-4d33-8833-e062f5cd01f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" Oct 30 21:04:07.063150 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:07.063122 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m62ft\" (UniqueName: \"kubernetes.io/projected/797a8c75-fb91-4d33-8833-e062f5cd01f0-kube-api-access-m62ft\") pod \"maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq\" (UID: \"797a8c75-fb91-4d33-8833-e062f5cd01f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" Oct 30 21:04:07.126610 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:07.126583 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" Oct 30 21:04:07.498351 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:07.498324 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq"] Oct 30 21:04:07.498608 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:04:07.498581 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797a8c75_fb91_4d33_8833_e062f5cd01f0.slice/crio-25df88775572e82745cca7d929b507fe0990fc18121706da9ea2dcc8f6a15ca8 WatchSource:0}: Error finding container 25df88775572e82745cca7d929b507fe0990fc18121706da9ea2dcc8f6a15ca8: Status 404 returned error can't find the container with id 25df88775572e82745cca7d929b507fe0990fc18121706da9ea2dcc8f6a15ca8 Oct 30 21:04:07.823632 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:07.823545 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" event={"ID":"797a8c75-fb91-4d33-8833-e062f5cd01f0","Type":"ContainerStarted","Data":"25df88775572e82745cca7d929b507fe0990fc18121706da9ea2dcc8f6a15ca8"} Oct 30 21:04:08.832439 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:08.832406 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" event={"ID":"797a8c75-fb91-4d33-8833-e062f5cd01f0","Type":"ContainerStarted","Data":"acaf7f0db5243bad928953775fd3c2cc3c1d2016da5566d3973ab49f9f22c672"} Oct 30 21:04:08.832879 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:08.832509 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" Oct 30 21:04:08.852600 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:08.852542 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" podStartSLOduration=2.219844576 podStartE2EDuration="2.852528861s" totalImagesPullingTime="632.684285ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:08.850562304 +0000 UTC m=+1554.030458810" watchObservedRunningTime="2026-10-30 21:04:08.852528861 +0000 UTC m=+1554.032425405" Oct 30 21:04:09.936374 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:09.936336 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w"] Oct 30 21:04:09.936973 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:09.936703 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" podUID="75e137a8-1096-4138-9e30-2cce04668543" containerName="maas-api" containerID="cri-o://a922bd85a6f64e9f2da7907745f2a61a5448f3d7b196ff26f0033d68e6009461" gracePeriod=30 Oct 30 21:04:10.014540 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.014507 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755"] Oct 30 21:04:10.014745 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.014719 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755" podUID="356460c8-2576-4249-8ba3-6efc7c78d124" containerName="payload-processing" containerID="cri-o://9e994a582be888497386789eb0c7da332706f94258db2f331d292a8c791bb1a4" gracePeriod=30 Oct 30 21:04:10.041613 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.041584 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq"] Oct 30 21:04:10.041877 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.041849 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq" podUID="a5fd2739-f2e8-48d3-8d56-e15367a9c620" containerName="payload-pre-processing" containerID="cri-o://fc0b95d610755d032a4807081b1a9ba38856ff721882e8c2af430202d603631c" gracePeriod=30 Oct 30 21:04:10.577246 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.576917 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" Oct 30 21:04:10.655499 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.654253 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e137a8-1096-4138-9e30-2cce04668543-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e137a8-1096-4138-9e30-2cce04668543-openshift-service-ca\") pod \"75e137a8-1096-4138-9e30-2cce04668543\" (UID: \"75e137a8-1096-4138-9e30-2cce04668543\") " Oct 30 21:04:10.655499 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.654386 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e137a8-1096-4138-9e30-2cce04668543-kube-api-access-xll9v\" (UniqueName: \"kubernetes.io/projected/75e137a8-1096-4138-9e30-2cce04668543-kube-api-access-xll9v\") pod \"75e137a8-1096-4138-9e30-2cce04668543\" (UID: \"75e137a8-1096-4138-9e30-2cce04668543\") " Oct 30 21:04:10.655499 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.654489 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75e137a8-1096-4138-9e30-2cce04668543-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e137a8-1096-4138-9e30-2cce04668543-maas-api-tls\") pod \"75e137a8-1096-4138-9e30-2cce04668543\" (UID: \"75e137a8-1096-4138-9e30-2cce04668543\") " Oct 30 21:04:10.655499 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.654713 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75e137a8-1096-4138-9e30-2cce04668543-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e137a8-1096-4138-9e30-2cce04668543-metrics-certs\") pod \"75e137a8-1096-4138-9e30-2cce04668543\" (UID: \"75e137a8-1096-4138-9e30-2cce04668543\") " Oct 30 21:04:10.655873 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.655627 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e137a8-1096-4138-9e30-2cce04668543-openshift-service-ca" pod "75e137a8-1096-4138-9e30-2cce04668543" (UID: "75e137a8-1096-4138-9e30-2cce04668543"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:04:10.658597 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.658555 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e137a8-1096-4138-9e30-2cce04668543-metrics-certs" pod "75e137a8-1096-4138-9e30-2cce04668543" (UID: "75e137a8-1096-4138-9e30-2cce04668543"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:04:10.658977 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.658957 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e137a8-1096-4138-9e30-2cce04668543-maas-api-tls" pod "75e137a8-1096-4138-9e30-2cce04668543" (UID: "75e137a8-1096-4138-9e30-2cce04668543"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:04:10.660227 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.660183 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e137a8-1096-4138-9e30-2cce04668543-kube-api-access-xll9v" pod "75e137a8-1096-4138-9e30-2cce04668543" (UID: "75e137a8-1096-4138-9e30-2cce04668543"). InnerVolumeSpecName "kube-api-access-xll9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:10.702496 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.702474 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755" Oct 30 21:04:10.705002 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.704981 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq" Oct 30 21:04:10.778838 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.772049 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5fd2739-f2e8-48d3-8d56-e15367a9c620-kube-api-access-j8497\" (UniqueName: \"kubernetes.io/projected/a5fd2739-f2e8-48d3-8d56-e15367a9c620-kube-api-access-j8497\") pod \"a5fd2739-f2e8-48d3-8d56-e15367a9c620\" (UID: \"a5fd2739-f2e8-48d3-8d56-e15367a9c620\") " Oct 30 21:04:10.778838 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.772562 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5fd2739-f2e8-48d3-8d56-e15367a9c620-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5fd2739-f2e8-48d3-8d56-e15367a9c620-plugins-config-volume\") pod \"a5fd2739-f2e8-48d3-8d56-e15367a9c620\" (UID: \"a5fd2739-f2e8-48d3-8d56-e15367a9c620\") " Oct 30 21:04:10.778838 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.772612 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/356460c8-2576-4249-8ba3-6efc7c78d124-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356460c8-2576-4249-8ba3-6efc7c78d124-plugins-config-volume\") pod \"356460c8-2576-4249-8ba3-6efc7c78d124\" (UID: \"356460c8-2576-4249-8ba3-6efc7c78d124\") " Oct 30 21:04:10.778838 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.772904 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/356460c8-2576-4249-8ba3-6efc7c78d124-kube-api-access-g9m2g\" (UniqueName: \"kubernetes.io/projected/356460c8-2576-4249-8ba3-6efc7c78d124-kube-api-access-g9m2g\") pod \"356460c8-2576-4249-8ba3-6efc7c78d124\" (UID: \"356460c8-2576-4249-8ba3-6efc7c78d124\") " Oct 30 21:04:10.778838 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.774431 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xll9v\" (UniqueName: \"kubernetes.io/projected/75e137a8-1096-4138-9e30-2cce04668543-kube-api-access-xll9v\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:10.778838 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.774463 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75e137a8-1096-4138-9e30-2cce04668543-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:10.778838 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.774478 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75e137a8-1096-4138-9e30-2cce04668543-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:10.778838 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.774493 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e137a8-1096-4138-9e30-2cce04668543-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:10.778838 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.777157 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/356460c8-2576-4249-8ba3-6efc7c78d124-plugins-config-volume" pod "356460c8-2576-4249-8ba3-6efc7c78d124" (UID: "356460c8-2576-4249-8ba3-6efc7c78d124"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:04:10.792822 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.790212 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5fd2739-f2e8-48d3-8d56-e15367a9c620-plugins-config-volume" pod "a5fd2739-f2e8-48d3-8d56-e15367a9c620" (UID: "a5fd2739-f2e8-48d3-8d56-e15367a9c620"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:04:10.827793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.821415 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fd2739-f2e8-48d3-8d56-e15367a9c620-kube-api-access-j8497" pod "a5fd2739-f2e8-48d3-8d56-e15367a9c620" (UID: "a5fd2739-f2e8-48d3-8d56-e15367a9c620"). InnerVolumeSpecName "kube-api-access-j8497". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:10.827793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.824056 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/356460c8-2576-4249-8ba3-6efc7c78d124-kube-api-access-g9m2g" pod "356460c8-2576-4249-8ba3-6efc7c78d124" (UID: "356460c8-2576-4249-8ba3-6efc7c78d124"). InnerVolumeSpecName "kube-api-access-g9m2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:10.873398 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.873369 4848 generic.go:356] "Generic (PLEG): container finished" podID="75e137a8-1096-4138-9e30-2cce04668543" containerID="a922bd85a6f64e9f2da7907745f2a61a5448f3d7b196ff26f0033d68e6009461" exitCode=0 Oct 30 21:04:10.873524 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.873454 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" event={"ID":"75e137a8-1096-4138-9e30-2cce04668543","Type":"ContainerDied","Data":"a922bd85a6f64e9f2da7907745f2a61a5448f3d7b196ff26f0033d68e6009461"} Oct 30 21:04:10.873524 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.873496 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" event={"ID":"75e137a8-1096-4138-9e30-2cce04668543","Type":"ContainerDied","Data":"7bc275d243e35737d56010fc880bd75497c92d88ebf0b23f627a04af15cf1dfa"} Oct 30 21:04:10.873524 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.873518 4848 scope.go:122] "RemoveContainer" containerID="a922bd85a6f64e9f2da7907745f2a61a5448f3d7b196ff26f0033d68e6009461" Oct 30 21:04:10.873702 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.873639 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w" Oct 30 21:04:10.876359 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.876327 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9m2g\" (UniqueName: \"kubernetes.io/projected/356460c8-2576-4249-8ba3-6efc7c78d124-kube-api-access-g9m2g\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:10.876471 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.876362 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8497\" (UniqueName: \"kubernetes.io/projected/a5fd2739-f2e8-48d3-8d56-e15367a9c620-kube-api-access-j8497\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:10.876471 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.876379 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5fd2739-f2e8-48d3-8d56-e15367a9c620-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:10.876471 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.876395 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/356460c8-2576-4249-8ba3-6efc7c78d124-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:10.881815 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.879398 4848 generic.go:356] "Generic (PLEG): container finished" podID="356460c8-2576-4249-8ba3-6efc7c78d124" containerID="9e994a582be888497386789eb0c7da332706f94258db2f331d292a8c791bb1a4" exitCode=0 Oct 30 21:04:10.881815 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.879478 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755" event={"ID":"356460c8-2576-4249-8ba3-6efc7c78d124","Type":"ContainerDied","Data":"9e994a582be888497386789eb0c7da332706f94258db2f331d292a8c791bb1a4"} Oct 30 21:04:10.881815 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.879520 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755" event={"ID":"356460c8-2576-4249-8ba3-6efc7c78d124","Type":"ContainerDied","Data":"609742c1564df6b219535a822092426a7e091f0dca8a976544096d8ca98e616d"} Oct 30 21:04:10.881815 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.879593 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755" Oct 30 21:04:10.893839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.887932 4848 generic.go:356] "Generic (PLEG): container finished" podID="a5fd2739-f2e8-48d3-8d56-e15367a9c620" containerID="fc0b95d610755d032a4807081b1a9ba38856ff721882e8c2af430202d603631c" exitCode=0 Oct 30 21:04:10.893839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.888045 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq" event={"ID":"a5fd2739-f2e8-48d3-8d56-e15367a9c620","Type":"ContainerDied","Data":"fc0b95d610755d032a4807081b1a9ba38856ff721882e8c2af430202d603631c"} Oct 30 21:04:10.893839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.888089 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq" event={"ID":"a5fd2739-f2e8-48d3-8d56-e15367a9c620","Type":"ContainerDied","Data":"b79309491ec7f0b60844f92a5140ca9254f283dcb005b58961323c15d66e201b"} Oct 30 21:04:10.893839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.888152 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq" Oct 30 21:04:10.900338 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.900310 4848 scope.go:122] "RemoveContainer" containerID="a922bd85a6f64e9f2da7907745f2a61a5448f3d7b196ff26f0033d68e6009461" Oct 30 21:04:10.900850 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:04:10.900734 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a922bd85a6f64e9f2da7907745f2a61a5448f3d7b196ff26f0033d68e6009461\": container with ID starting with a922bd85a6f64e9f2da7907745f2a61a5448f3d7b196ff26f0033d68e6009461 not found: ID does not exist" containerID="a922bd85a6f64e9f2da7907745f2a61a5448f3d7b196ff26f0033d68e6009461" Oct 30 21:04:10.900850 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.900835 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a922bd85a6f64e9f2da7907745f2a61a5448f3d7b196ff26f0033d68e6009461"} err="failed to get container status \"a922bd85a6f64e9f2da7907745f2a61a5448f3d7b196ff26f0033d68e6009461\": rpc error: code = NotFound desc = could not find container \"a922bd85a6f64e9f2da7907745f2a61a5448f3d7b196ff26f0033d68e6009461\": container with ID starting with a922bd85a6f64e9f2da7907745f2a61a5448f3d7b196ff26f0033d68e6009461 not found: ID does not exist" Oct 30 21:04:10.900996 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.900861 4848 scope.go:122] "RemoveContainer" containerID="9e994a582be888497386789eb0c7da332706f94258db2f331d292a8c791bb1a4" Oct 30 21:04:10.911631 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.911588 4848 scope.go:122] "RemoveContainer" containerID="9e994a582be888497386789eb0c7da332706f94258db2f331d292a8c791bb1a4" Oct 30 21:04:10.911963 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:04:10.911909 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e994a582be888497386789eb0c7da332706f94258db2f331d292a8c791bb1a4\": container with ID starting with 9e994a582be888497386789eb0c7da332706f94258db2f331d292a8c791bb1a4 not found: ID does not exist" containerID="9e994a582be888497386789eb0c7da332706f94258db2f331d292a8c791bb1a4" Oct 30 21:04:10.911963 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.911937 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e994a582be888497386789eb0c7da332706f94258db2f331d292a8c791bb1a4"} err="failed to get container status \"9e994a582be888497386789eb0c7da332706f94258db2f331d292a8c791bb1a4\": rpc error: code = NotFound desc = could not find container \"9e994a582be888497386789eb0c7da332706f94258db2f331d292a8c791bb1a4\": container with ID starting with 9e994a582be888497386789eb0c7da332706f94258db2f331d292a8c791bb1a4 not found: ID does not exist" Oct 30 21:04:10.912124 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.911975 4848 scope.go:122] "RemoveContainer" containerID="fc0b95d610755d032a4807081b1a9ba38856ff721882e8c2af430202d603631c" Oct 30 21:04:10.918163 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.918140 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w"] Oct 30 21:04:10.928827 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.928797 4848 scope.go:122] "RemoveContainer" containerID="fc0b95d610755d032a4807081b1a9ba38856ff721882e8c2af430202d603631c" Oct 30 21:04:10.929430 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:04:10.929402 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc0b95d610755d032a4807081b1a9ba38856ff721882e8c2af430202d603631c\": container with ID starting with fc0b95d610755d032a4807081b1a9ba38856ff721882e8c2af430202d603631c not found: ID does not exist" containerID="fc0b95d610755d032a4807081b1a9ba38856ff721882e8c2af430202d603631c" Oct 30 21:04:10.929534 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.929439 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc0b95d610755d032a4807081b1a9ba38856ff721882e8c2af430202d603631c"} err="failed to get container status \"fc0b95d610755d032a4807081b1a9ba38856ff721882e8c2af430202d603631c\": rpc error: code = NotFound desc = could not find container \"fc0b95d610755d032a4807081b1a9ba38856ff721882e8c2af430202d603631c\": container with ID starting with fc0b95d610755d032a4807081b1a9ba38856ff721882e8c2af430202d603631c not found: ID does not exist" Oct 30 21:04:10.943393 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.943372 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7c7768-6db4998595-dv69w"] Oct 30 21:04:10.943727 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.943402 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755"] Oct 30 21:04:10.943727 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.943416 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7c7768-7c46496cdd-bg755"] Oct 30 21:04:10.954743 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.954711 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq"] Oct 30 21:04:10.958266 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:10.958246 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-7c7768-5bcb6cb7f5-b5nlq"] Oct 30 21:04:12.935171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:12.935126 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="356460c8-2576-4249-8ba3-6efc7c78d124" path="/var/lib/kubelet/pods/356460c8-2576-4249-8ba3-6efc7c78d124/volumes" Oct 30 21:04:12.935895 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:12.935875 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e137a8-1096-4138-9e30-2cce04668543" path="/var/lib/kubelet/pods/75e137a8-1096-4138-9e30-2cce04668543/volumes" Oct 30 21:04:12.936452 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:12.936433 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5fd2739-f2e8-48d3-8d56-e15367a9c620" path="/var/lib/kubelet/pods/a5fd2739-f2e8-48d3-8d56-e15367a9c620/volumes" Oct 30 21:04:14.863528 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:14.863436 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" Oct 30 21:04:16.785260 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:16.785225 4848 scope.go:122] "RemoveContainer" containerID="072192f1b8a41c1fa6a026efc64d597a75285d036e5aac6af8f32ac70860f210" Oct 30 21:04:17.834811 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:17.832074 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6"] Oct 30 21:04:17.834811 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:17.832358 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" podUID="3c8ef112-9213-4f15-959c-63e09bb7faf5" containerName="istio-proxy" containerID="cri-o://9b779f3d9283047cc8faa6fd5db7216caadf9dcf36a2c1a626dc19832445f120" gracePeriod=30 Oct 30 21:04:19.914633 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:19.914594 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs"] Oct 30 21:04:19.915662 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:19.915640 4848 cpu_manager.go:414] "containerMap: removing container" podUID="75e137a8-1096-4138-9e30-2cce04668543" containerName="maas-api" Oct 30 21:04:19.915709 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:19.915666 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e137a8-1096-4138-9e30-2cce04668543" containerName="maas-api" Oct 30 21:04:19.915709 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:19.915693 4848 cpu_manager.go:414] "containerMap: removing container" podUID="a5fd2739-f2e8-48d3-8d56-e15367a9c620" containerName="payload-pre-processing" Oct 30 21:04:19.915709 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:19.915701 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5fd2739-f2e8-48d3-8d56-e15367a9c620" containerName="payload-pre-processing" Oct 30 21:04:19.915847 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:19.915728 4848 cpu_manager.go:414] "containerMap: removing container" podUID="356460c8-2576-4249-8ba3-6efc7c78d124" containerName="payload-processing" Oct 30 21:04:19.915847 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:19.915736 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="356460c8-2576-4249-8ba3-6efc7c78d124" containerName="payload-processing" Oct 30 21:04:19.915971 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:19.915955 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="356460c8-2576-4249-8ba3-6efc7c78d124" containerName="payload-processing" Oct 30 21:04:19.916006 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:19.915989 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e137a8-1096-4138-9e30-2cce04668543" containerName="maas-api" Oct 30 21:04:19.916006 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:19.916003 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5fd2739-f2e8-48d3-8d56-e15367a9c620" containerName="payload-pre-processing" Oct 30 21:04:19.922206 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:19.922192 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:19.935943 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:19.935913 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-89d993-openshift-default-dockercfg-tx9c6\"" Oct 30 21:04:19.936214 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:19.936191 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs"] Oct 30 21:04:20.083788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.053512 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-data\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.102188 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.102151 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-podinfo\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.102483 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.102465 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae7f986-ff45-410c-8301-3e23132e0bc5-istiod-ca-cert\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.102718 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.102705 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-workload-socket\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.106792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.102922 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-workload-certs\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.107147 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.107077 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht7jg\" (UniqueName: \"kubernetes.io/projected/eae7f986-ff45-410c-8301-3e23132e0bc5-kube-api-access-ht7jg\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.124723 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.107372 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-envoy\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.125171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.125134 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-token\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.125288 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.125208 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-credential-socket\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.227787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.226781 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-workload-socket\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.227787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.226897 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-workload-certs\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.227787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.226934 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht7jg\" (UniqueName: \"kubernetes.io/projected/eae7f986-ff45-410c-8301-3e23132e0bc5-kube-api-access-ht7jg\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.227787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.226983 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-envoy\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.227787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.227050 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-token\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.227787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.227091 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-credential-socket\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.227787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.227134 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-data\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.227787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.227174 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-podinfo\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.227787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.227241 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae7f986-ff45-410c-8301-3e23132e0bc5-istiod-ca-cert\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.228367 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.228066 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae7f986-ff45-410c-8301-3e23132e0bc5-istiod-ca-cert\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.228367 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.228343 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-workload-socket\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.240821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.228576 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-workload-certs\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.240821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.234303 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-credential-socket\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.240821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.235047 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-data\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.253845 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.252232 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-token\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.253845 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.252499 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-podinfo\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.272789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.267759 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-envoy\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.300514 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.298020 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht7jg\" (UniqueName: \"kubernetes.io/projected/eae7f986-ff45-410c-8301-3e23132e0bc5-kube-api-access-ht7jg\") pod \"e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.537747 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.537646 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:20.880621 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.880210 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs"] Oct 30 21:04:20.935293 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.934490 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:04:20.935293 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.934590 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:04:20.935293 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:20.934631 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:04:21.182314 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:21.182280 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" event={"ID":"eae7f986-ff45-410c-8301-3e23132e0bc5","Type":"ContainerStarted","Data":"2d70a49c564e94d908b5e3f67166c4cf650bd120f2cb1960410f0e6500def992"} Oct 30 21:04:21.182593 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:21.182565 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" event={"ID":"eae7f986-ff45-410c-8301-3e23132e0bc5","Type":"ContainerStarted","Data":"d11e88875f100250c55397289d5d7a7d58c7a628cd939dfb4ed28572aa21c885"} Oct 30 21:04:21.210165 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:21.210078 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" podStartSLOduration=2.21006143 podStartE2EDuration="2.21006143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:21.204845681 +0000 UTC m=+1566.384742189" watchObservedRunningTime="2026-10-30 21:04:21.21006143 +0000 UTC m=+1566.389957936" Oct 30 21:04:21.537981 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:21.537907 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:22.191551 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:22.191519 4848 patch_prober.go:29] interesting pod/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:04:22.192025 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:22.191573 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" podUID="3c8ef112-9213-4f15-959c-63e09bb7faf5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:04:22.511392 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:04:22.511306 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:04:22.544421 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:22.544390 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:23.197099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.197066 4848 generic.go:356] "Generic (PLEG): container finished" podID="3c8ef112-9213-4f15-959c-63e09bb7faf5" containerID="9b779f3d9283047cc8faa6fd5db7216caadf9dcf36a2c1a626dc19832445f120" exitCode=0 Oct 30 21:04:23.197538 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.197138 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" event={"ID":"3c8ef112-9213-4f15-959c-63e09bb7faf5","Type":"ContainerDied","Data":"9b779f3d9283047cc8faa6fd5db7216caadf9dcf36a2c1a626dc19832445f120"} Oct 30 21:04:23.197538 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.197203 4848 scope.go:122] "RemoveContainer" containerID="15b5d23359bf9635b589a8cd2493ceec0ed416cbd845253702a20a9c66ca9c59" Oct 30 21:04:23.197538 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.197462 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:23.198632 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.198610 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:04:23.540650 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.540621 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 21:04:23.678501 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.678466 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-workload-certs\") pod \"3c8ef112-9213-4f15-959c-63e09bb7faf5\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " Oct 30 21:04:23.678704 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.678541 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-envoy\") pod \"3c8ef112-9213-4f15-959c-63e09bb7faf5\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " Oct 30 21:04:23.678704 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.678574 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-credential-socket\") pod \"3c8ef112-9213-4f15-959c-63e09bb7faf5\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " Oct 30 21:04:23.678704 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.678659 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8ef112-9213-4f15-959c-63e09bb7faf5-kube-api-access-tfhm7\" (UniqueName: \"kubernetes.io/projected/3c8ef112-9213-4f15-959c-63e09bb7faf5-kube-api-access-tfhm7\") pod \"3c8ef112-9213-4f15-959c-63e09bb7faf5\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " Oct 30 21:04:23.678986 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.678747 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-podinfo\") pod \"3c8ef112-9213-4f15-959c-63e09bb7faf5\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " Oct 30 21:04:23.678986 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.678840 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-workload-socket\") pod \"3c8ef112-9213-4f15-959c-63e09bb7faf5\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " Oct 30 21:04:23.678986 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.678849 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-workload-certs" pod "3c8ef112-9213-4f15-959c-63e09bb7faf5" (UID: "3c8ef112-9213-4f15-959c-63e09bb7faf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:23.678986 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.678885 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-token\" (UniqueName: \"kubernetes.io/projected/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-token\") pod \"3c8ef112-9213-4f15-959c-63e09bb7faf5\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " Oct 30 21:04:23.679221 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.679007 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c8ef112-9213-4f15-959c-63e09bb7faf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8ef112-9213-4f15-959c-63e09bb7faf5-istiod-ca-cert\") pod \"3c8ef112-9213-4f15-959c-63e09bb7faf5\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " Oct 30 21:04:23.679221 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.679027 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-credential-socket" pod "3c8ef112-9213-4f15-959c-63e09bb7faf5" (UID: "3c8ef112-9213-4f15-959c-63e09bb7faf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:23.679221 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.679096 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-data\") pod \"3c8ef112-9213-4f15-959c-63e09bb7faf5\" (UID: \"3c8ef112-9213-4f15-959c-63e09bb7faf5\") " Oct 30 21:04:23.679748 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.679701 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8ef112-9213-4f15-959c-63e09bb7faf5-istiod-ca-cert" pod "3c8ef112-9213-4f15-959c-63e09bb7faf5" (UID: "3c8ef112-9213-4f15-959c-63e09bb7faf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:04:23.679982 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.679957 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-workload-socket" pod "3c8ef112-9213-4f15-959c-63e09bb7faf5" (UID: "3c8ef112-9213-4f15-959c-63e09bb7faf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:23.680228 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.680204 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:23.680309 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.680255 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8ef112-9213-4f15-959c-63e09bb7faf5-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:23.680309 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.680272 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:23.680309 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.680288 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:23.680526 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.680507 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-data" pod "3c8ef112-9213-4f15-959c-63e09bb7faf5" (UID: "3c8ef112-9213-4f15-959c-63e09bb7faf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:23.681477 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.681452 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-token" pod "3c8ef112-9213-4f15-959c-63e09bb7faf5" (UID: "3c8ef112-9213-4f15-959c-63e09bb7faf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:23.681640 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.681614 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-envoy" pod "3c8ef112-9213-4f15-959c-63e09bb7faf5" (UID: "3c8ef112-9213-4f15-959c-63e09bb7faf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:23.682247 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.682221 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8ef112-9213-4f15-959c-63e09bb7faf5-kube-api-access-tfhm7" pod "3c8ef112-9213-4f15-959c-63e09bb7faf5" (UID: "3c8ef112-9213-4f15-959c-63e09bb7faf5"). InnerVolumeSpecName "kube-api-access-tfhm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:23.682510 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.682492 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-podinfo" pod "3c8ef112-9213-4f15-959c-63e09bb7faf5" (UID: "3c8ef112-9213-4f15-959c-63e09bb7faf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:04:23.781973 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.781888 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:23.781973 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.781917 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfhm7\" (UniqueName: \"kubernetes.io/projected/3c8ef112-9213-4f15-959c-63e09bb7faf5-kube-api-access-tfhm7\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:23.781973 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.781928 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:23.781973 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.781937 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:23.781973 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:23.781946 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8ef112-9213-4f15-959c-63e09bb7faf5-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:04:24.205126 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.205096 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" Oct 30 21:04:24.205601 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.205138 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6" event={"ID":"3c8ef112-9213-4f15-959c-63e09bb7faf5","Type":"ContainerDied","Data":"fd1e6b9092af2c97f8fcf04f2830cf0d5239a02dd3654bac73d5cdd784f5e473"} Oct 30 21:04:24.205601 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.205198 4848 scope.go:122] "RemoveContainer" containerID="9b779f3d9283047cc8faa6fd5db7216caadf9dcf36a2c1a626dc19832445f120" Oct 30 21:04:24.231297 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.231265 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6"] Oct 30 21:04:24.239453 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.239426 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-7c7768-openshift-default-6576b8554f-7vtq6"] Oct 30 21:04:24.745231 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.745195 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc"] Oct 30 21:04:24.745913 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.745894 4848 cpu_manager.go:414] "containerMap: removing container" podUID="3c8ef112-9213-4f15-959c-63e09bb7faf5" containerName="istio-proxy" Oct 30 21:04:24.745913 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.745913 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c8ef112-9213-4f15-959c-63e09bb7faf5" containerName="istio-proxy" Oct 30 21:04:24.746130 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.746116 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c8ef112-9213-4f15-959c-63e09bb7faf5" containerName="istio-proxy" Oct 30 21:04:24.750989 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.750967 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc" Oct 30 21:04:24.755312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.755295 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-89d993\"" Oct 30 21:04:24.768802 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.768777 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc"] Oct 30 21:04:24.806259 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.806229 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72"] Oct 30 21:04:24.806968 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.806948 4848 cpu_manager.go:414] "containerMap: removing container" podUID="3c8ef112-9213-4f15-959c-63e09bb7faf5" containerName="istio-proxy" Oct 30 21:04:24.806968 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.806968 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c8ef112-9213-4f15-959c-63e09bb7faf5" containerName="istio-proxy" Oct 30 21:04:24.807198 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.807183 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c8ef112-9213-4f15-959c-63e09bb7faf5" containerName="istio-proxy" Oct 30 21:04:24.810718 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.810697 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72" Oct 30 21:04:24.821717 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.821688 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4zgs\" (UniqueName: \"kubernetes.io/projected/711bf26e-5e4f-4b03-93c2-5302e22b543d-kube-api-access-t4zgs\") pod \"payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc\" (UID: \"711bf26e-5e4f-4b03-93c2-5302e22b543d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc" Oct 30 21:04:24.821877 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.821754 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711bf26e-5e4f-4b03-93c2-5302e22b543d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc\" (UID: \"711bf26e-5e4f-4b03-93c2-5302e22b543d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc" Oct 30 21:04:24.825189 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.823598 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-89d993-dockercfg-z2vqj\"" Oct 30 21:04:24.829715 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:24.829686 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72"] Oct 30 21:04:25.130110 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:25.098016 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t926g\" (UniqueName: \"kubernetes.io/projected/fd1bad1a-2f48-4b2e-801d-ad3762be02c8-kube-api-access-t926g\") pod \"payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72\" (UID: \"fd1bad1a-2f48-4b2e-801d-ad3762be02c8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72" Oct 30 21:04:25.130110 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:25.098144 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd1bad1a-2f48-4b2e-801d-ad3762be02c8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72\" (UID: \"fd1bad1a-2f48-4b2e-801d-ad3762be02c8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72" Oct 30 21:04:25.130110 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:25.098245 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4zgs\" (UniqueName: \"kubernetes.io/projected/711bf26e-5e4f-4b03-93c2-5302e22b543d-kube-api-access-t4zgs\") pod \"payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc\" (UID: \"711bf26e-5e4f-4b03-93c2-5302e22b543d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc" Oct 30 21:04:25.130110 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:25.098298 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711bf26e-5e4f-4b03-93c2-5302e22b543d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc\" (UID: \"711bf26e-5e4f-4b03-93c2-5302e22b543d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc" Oct 30 21:04:25.130110 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:25.099396 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711bf26e-5e4f-4b03-93c2-5302e22b543d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc\" (UID: \"711bf26e-5e4f-4b03-93c2-5302e22b543d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc" Oct 30 21:04:25.177662 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:25.155083 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4zgs\" (UniqueName: \"kubernetes.io/projected/711bf26e-5e4f-4b03-93c2-5302e22b543d-kube-api-access-t4zgs\") pod \"payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc\" (UID: \"711bf26e-5e4f-4b03-93c2-5302e22b543d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc" Oct 30 21:04:25.181933 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:25.181904 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c8ef112-9213-4f15-959c-63e09bb7faf5" path="/var/lib/kubelet/pods/3c8ef112-9213-4f15-959c-63e09bb7faf5/volumes" Oct 30 21:04:25.200337 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:25.200299 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd1bad1a-2f48-4b2e-801d-ad3762be02c8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72\" (UID: \"fd1bad1a-2f48-4b2e-801d-ad3762be02c8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72" Oct 30 21:04:25.200648 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:25.200627 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t926g\" (UniqueName: \"kubernetes.io/projected/fd1bad1a-2f48-4b2e-801d-ad3762be02c8-kube-api-access-t926g\") pod \"payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72\" (UID: \"fd1bad1a-2f48-4b2e-801d-ad3762be02c8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72" Oct 30 21:04:25.201121 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:25.201099 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd1bad1a-2f48-4b2e-801d-ad3762be02c8-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72\" (UID: \"fd1bad1a-2f48-4b2e-801d-ad3762be02c8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72" Oct 30 21:04:25.210160 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:25.210138 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t926g\" (UniqueName: \"kubernetes.io/projected/fd1bad1a-2f48-4b2e-801d-ad3762be02c8-kube-api-access-t926g\") pod \"payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72\" (UID: \"fd1bad1a-2f48-4b2e-801d-ad3762be02c8\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72" Oct 30 21:04:25.365463 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:25.365424 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc" Oct 30 21:04:25.433732 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:25.433703 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72" Oct 30 21:04:25.697122 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:25.697082 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc"] Oct 30 21:04:25.712280 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:04:25.712243 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711bf26e_5e4f_4b03_93c2_5302e22b543d.slice/crio-b220cde6bc2aa1d6ca96b5d329d804224654b98b177f2e02fc1e38caf6f554a0 WatchSource:0}: Error finding container b220cde6bc2aa1d6ca96b5d329d804224654b98b177f2e02fc1e38caf6f554a0: Status 404 returned error can't find the container with id b220cde6bc2aa1d6ca96b5d329d804224654b98b177f2e02fc1e38caf6f554a0 Oct 30 21:04:25.736312 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:04:25.736285 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1bad1a_2f48_4b2e_801d_ad3762be02c8.slice/crio-0cbe0f8529351bf415e028c8af13c5a39acbdbac3e99fecacd957f3e8df9d8de WatchSource:0}: Error finding container 0cbe0f8529351bf415e028c8af13c5a39acbdbac3e99fecacd957f3e8df9d8de: Status 404 returned error can't find the container with id 0cbe0f8529351bf415e028c8af13c5a39acbdbac3e99fecacd957f3e8df9d8de Oct 30 21:04:25.736647 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:25.736627 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72"] Oct 30 21:04:26.222917 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:26.222877 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72" event={"ID":"fd1bad1a-2f48-4b2e-801d-ad3762be02c8","Type":"ContainerStarted","Data":"32cd53445921a9c54c082566c8a2a58114f3e457a82c649381c5766a9b057513"} Oct 30 21:04:26.223373 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:26.222951 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72" event={"ID":"fd1bad1a-2f48-4b2e-801d-ad3762be02c8","Type":"ContainerStarted","Data":"0cbe0f8529351bf415e028c8af13c5a39acbdbac3e99fecacd957f3e8df9d8de"} Oct 30 21:04:26.223373 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:26.223106 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72" Oct 30 21:04:26.225074 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:26.225043 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc" event={"ID":"711bf26e-5e4f-4b03-93c2-5302e22b543d","Type":"ContainerStarted","Data":"3212d060a18b1c6e5606169a0485e2f8a6660349c3e3cf2e6557effdbaaf73dc"} Oct 30 21:04:26.225207 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:26.225100 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc" event={"ID":"711bf26e-5e4f-4b03-93c2-5302e22b543d","Type":"ContainerStarted","Data":"b220cde6bc2aa1d6ca96b5d329d804224654b98b177f2e02fc1e38caf6f554a0"} Oct 30 21:04:26.225207 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:26.225134 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc" Oct 30 21:04:26.245740 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:26.245670 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72" podStartSLOduration=2.245652575 podStartE2EDuration="2.245652575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:26.240871155 +0000 UTC m=+1571.420767662" watchObservedRunningTime="2026-10-30 21:04:26.245652575 +0000 UTC m=+1571.425549083" Oct 30 21:04:26.262984 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:26.262924 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc" podStartSLOduration=2.262909467 podStartE2EDuration="2.262909467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:26.25922516 +0000 UTC m=+1571.439121666" watchObservedRunningTime="2026-10-30 21:04:26.262909467 +0000 UTC m=+1571.442806048" Oct 30 21:04:27.452092 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.452056 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv"] Oct 30 21:04:27.457486 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.457462 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:27.465365 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.465333 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-3a65ec\"/\"kube-root-ca.crt\"" Oct 30 21:04:27.467733 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.466242 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-3a65ec\"/\"openshift-service-ca.crt\"" Oct 30 21:04:27.467733 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.466349 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-3a65ec\"/\"auth-test-model-3a65ec-kserve-self-signed-certs\"" Oct 30 21:04:27.467733 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.466396 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-3a65ec\"/\"default-dockercfg-lgpnn\"" Oct 30 21:04:27.467733 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.466740 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv"] Oct 30 21:04:27.560820 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.560754 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qp8h\" (UniqueName: \"kubernetes.io/projected/9c2f7980-610f-44e9-a586-b48bcf05d315-kube-api-access-2qp8h\") pod \"auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:27.561007 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.560951 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-dshm\") pod \"auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:27.561007 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.560992 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-model-cache\") pod \"auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:27.561126 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.561111 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2f7980-610f-44e9-a586-b48bcf05d315-tls-certs\") pod \"auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:27.561199 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.561184 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-tmp-dir\") pod \"auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:27.561272 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.561258 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-home\") pod \"auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:27.679790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.663756 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2f7980-610f-44e9-a586-b48bcf05d315-tls-certs\") pod \"auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:27.679790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.664002 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-tmp-dir\") pod \"auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:27.679790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.664041 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-home\") pod \"auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:27.679790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.664149 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qp8h\" (UniqueName: \"kubernetes.io/projected/9c2f7980-610f-44e9-a586-b48bcf05d315-kube-api-access-2qp8h\") pod \"auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:27.679790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.664287 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-dshm\") pod \"auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:27.679790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.664337 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-model-cache\") pod \"auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:27.679790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.664860 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-model-cache\") pod \"auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:27.679790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.670110 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-home\") pod \"auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:27.679790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.670417 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-tmp-dir\") pod \"auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:27.679790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.674478 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2f7980-610f-44e9-a586-b48bcf05d315-tls-certs\") pod \"auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:27.694797 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.689909 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-dshm\") pod \"auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:27.698791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.697209 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qp8h\" (UniqueName: \"kubernetes.io/projected/9c2f7980-610f-44e9-a586-b48bcf05d315-kube-api-access-2qp8h\") pod \"auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:27.783293 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:27.782072 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:28.157142 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:28.157114 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv"] Oct 30 21:04:28.158943 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:04:28.158915 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c2f7980_610f_44e9_a586_b48bcf05d315.slice/crio-35d570ed30ea754aac750525bf3442234166b2185225e75f253024c9531c7498 WatchSource:0}: Error finding container 35d570ed30ea754aac750525bf3442234166b2185225e75f253024c9531c7498: Status 404 returned error can't find the container with id 35d570ed30ea754aac750525bf3442234166b2185225e75f253024c9531c7498 Oct 30 21:04:28.248835 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:28.248745 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" event={"ID":"9c2f7980-610f-44e9-a586-b48bcf05d315","Type":"ContainerStarted","Data":"d858179ca4afb292f2c7084d75a06f3105ba9f7ea6cc98f700a3333beb154318"} Oct 30 21:04:28.248835 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:28.248809 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" event={"ID":"9c2f7980-610f-44e9-a586-b48bcf05d315","Type":"ContainerStarted","Data":"35d570ed30ea754aac750525bf3442234166b2185225e75f253024c9531c7498"} Oct 30 21:04:28.268987 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:28.268931 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" podStartSLOduration=1.2689143920000001 podStartE2EDuration="1.268914392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:28.265699832 +0000 UTC m=+1573.445596345" watchObservedRunningTime="2026-10-30 21:04:28.268914392 +0000 UTC m=+1573.448810901" Oct 30 21:04:28.782508 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:28.782470 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:35.252447 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:35.252407 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45"] Oct 30 21:04:35.309828 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:35.303942 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45"] Oct 30 21:04:35.309828 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:35.304069 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" Oct 30 21:04:35.401889 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:35.355483 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-89d993-serving-cert\"" Oct 30 21:04:35.530456 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:35.509449 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94768319-93cc-4cc4-ae81-ab6568a965c4-metrics-certs\") pod \"maas-api-e2e-praxis-a-89d993-97b69c496-crq45\" (UID: \"94768319-93cc-4cc4-ae81-ab6568a965c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" Oct 30 21:04:35.530456 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:35.509579 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94768319-93cc-4cc4-ae81-ab6568a965c4-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-89d993-97b69c496-crq45\" (UID: \"94768319-93cc-4cc4-ae81-ab6568a965c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" Oct 30 21:04:35.530456 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:35.509637 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94768319-93cc-4cc4-ae81-ab6568a965c4-maas-api-tls\") pod \"maas-api-e2e-praxis-a-89d993-97b69c496-crq45\" (UID: \"94768319-93cc-4cc4-ae81-ab6568a965c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" Oct 30 21:04:35.530456 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:35.509743 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvx6p\" (UniqueName: \"kubernetes.io/projected/94768319-93cc-4cc4-ae81-ab6568a965c4-kube-api-access-cvx6p\") pod \"maas-api-e2e-praxis-a-89d993-97b69c496-crq45\" (UID: \"94768319-93cc-4cc4-ae81-ab6568a965c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" Oct 30 21:04:35.642806 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:35.642622 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvx6p\" (UniqueName: \"kubernetes.io/projected/94768319-93cc-4cc4-ae81-ab6568a965c4-kube-api-access-cvx6p\") pod \"maas-api-e2e-praxis-a-89d993-97b69c496-crq45\" (UID: \"94768319-93cc-4cc4-ae81-ab6568a965c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" Oct 30 21:04:35.642992 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:35.642878 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94768319-93cc-4cc4-ae81-ab6568a965c4-metrics-certs\") pod \"maas-api-e2e-praxis-a-89d993-97b69c496-crq45\" (UID: \"94768319-93cc-4cc4-ae81-ab6568a965c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" Oct 30 21:04:35.662793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:35.660064 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94768319-93cc-4cc4-ae81-ab6568a965c4-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-89d993-97b69c496-crq45\" (UID: \"94768319-93cc-4cc4-ae81-ab6568a965c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" Oct 30 21:04:35.662793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:35.660190 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94768319-93cc-4cc4-ae81-ab6568a965c4-maas-api-tls\") pod \"maas-api-e2e-praxis-a-89d993-97b69c496-crq45\" (UID: \"94768319-93cc-4cc4-ae81-ab6568a965c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" Oct 30 21:04:35.684787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:35.678718 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94768319-93cc-4cc4-ae81-ab6568a965c4-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-89d993-97b69c496-crq45\" (UID: \"94768319-93cc-4cc4-ae81-ab6568a965c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" Oct 30 21:04:35.686149 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:35.686121 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94768319-93cc-4cc4-ae81-ab6568a965c4-maas-api-tls\") pod \"maas-api-e2e-praxis-a-89d993-97b69c496-crq45\" (UID: \"94768319-93cc-4cc4-ae81-ab6568a965c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" Oct 30 21:04:35.687963 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:35.687938 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvx6p\" (UniqueName: \"kubernetes.io/projected/94768319-93cc-4cc4-ae81-ab6568a965c4-kube-api-access-cvx6p\") pod \"maas-api-e2e-praxis-a-89d993-97b69c496-crq45\" (UID: \"94768319-93cc-4cc4-ae81-ab6568a965c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" Oct 30 21:04:35.688821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:35.688799 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94768319-93cc-4cc4-ae81-ab6568a965c4-metrics-certs\") pod \"maas-api-e2e-praxis-a-89d993-97b69c496-crq45\" (UID: \"94768319-93cc-4cc4-ae81-ab6568a965c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" Oct 30 21:04:35.891857 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:35.891726 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" Oct 30 21:04:36.646822 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:36.646678 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45"] Oct 30 21:04:36.647438 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:04:36.647406 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94768319_93cc_4cc4_ae81_ab6568a965c4.slice/crio-c1ce210dcb8e1faf6fac459208212aee6bf7abbe807ee986592ce577418f14b2 WatchSource:0}: Error finding container c1ce210dcb8e1faf6fac459208212aee6bf7abbe807ee986592ce577418f14b2: Status 404 returned error can't find the container with id c1ce210dcb8e1faf6fac459208212aee6bf7abbe807ee986592ce577418f14b2 Oct 30 21:04:37.243057 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.242998 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72" Oct 30 21:04:37.243263 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.243167 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc" Oct 30 21:04:37.318913 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.318874 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx"] Oct 30 21:04:37.364576 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.364507 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx"] Oct 30 21:04:37.364787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.364653 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.369112 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.369094 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-e53c37-openshift-default-dockercfg-kkgwd\"" Oct 30 21:04:37.456557 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.456526 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-istio-envoy\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.456690 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.456631 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p659p\" (UniqueName: \"kubernetes.io/projected/d2715482-f623-43ef-93b8-c92d26626bc9-kube-api-access-p659p\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.456690 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.456673 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-workload-certs\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.456781 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.456753 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2715482-f623-43ef-93b8-c92d26626bc9-istiod-ca-cert\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.456826 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.456799 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-istio-data\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.456869 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.456828 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2715482-f623-43ef-93b8-c92d26626bc9-istio-podinfo\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.456908 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.456891 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-workload-socket\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.456944 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.456908 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-credential-socket\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.456944 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.456939 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2715482-f623-43ef-93b8-c92d26626bc9-istio-token\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.558274 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.558182 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p659p\" (UniqueName: \"kubernetes.io/projected/d2715482-f623-43ef-93b8-c92d26626bc9-kube-api-access-p659p\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.558274 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.558243 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-workload-certs\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.558505 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.558325 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2715482-f623-43ef-93b8-c92d26626bc9-istiod-ca-cert\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.558505 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.558364 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-istio-data\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.558615 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.558516 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2715482-f623-43ef-93b8-c92d26626bc9-istio-podinfo\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.558682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.558646 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-workload-socket\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.558739 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.558678 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-credential-socket\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.558739 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.558719 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2715482-f623-43ef-93b8-c92d26626bc9-istio-token\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.558739 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.558719 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-workload-certs\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.558950 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.558747 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-istio-data\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.558950 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.558878 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-istio-envoy\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.559056 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.559025 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-credential-socket\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.559168 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.559151 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2715482-f623-43ef-93b8-c92d26626bc9-istiod-ca-cert\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.559354 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.559328 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-workload-socket\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.560993 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.560972 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-istio-envoy\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.561224 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.561203 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2715482-f623-43ef-93b8-c92d26626bc9-istio-podinfo\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.570315 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.570287 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2715482-f623-43ef-93b8-c92d26626bc9-istio-token\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.570543 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.570519 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p659p\" (UniqueName: \"kubernetes.io/projected/d2715482-f623-43ef-93b8-c92d26626bc9-kube-api-access-p659p\") pod \"e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.594912 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.594877 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" event={"ID":"94768319-93cc-4cc4-ae81-ab6568a965c4","Type":"ContainerStarted","Data":"5f5d8a64c68300654efd39a789829c13a8b520e5814bd99cb886680676511662"} Oct 30 21:04:37.595099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.594933 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" event={"ID":"94768319-93cc-4cc4-ae81-ab6568a965c4","Type":"ContainerStarted","Data":"c1ce210dcb8e1faf6fac459208212aee6bf7abbe807ee986592ce577418f14b2"} Oct 30 21:04:37.595099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.594979 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" Oct 30 21:04:37.625792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.616286 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" podStartSLOduration=1.973152025 podStartE2EDuration="2.616263932s" totalImagesPullingTime="643.111907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:37.613303548 +0000 UTC m=+1582.793200054" watchObservedRunningTime="2026-10-30 21:04:37.616263932 +0000 UTC m=+1582.796160440" Oct 30 21:04:37.753795 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.751625 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:37.793663 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.792238 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:37.885663 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:37.885571 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:38.166465 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:04:38.166436 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2715482_f623_43ef_93b8_c92d26626bc9.slice/crio-c44fd859850cdd96e619729f8dab173480e9304650ecb2f254f986bf07243b9f WatchSource:0}: Error finding container c44fd859850cdd96e619729f8dab173480e9304650ecb2f254f986bf07243b9f: Status 404 returned error can't find the container with id c44fd859850cdd96e619729f8dab173480e9304650ecb2f254f986bf07243b9f Oct 30 21:04:38.167897 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:38.167872 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx"] Oct 30 21:04:38.169489 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:38.169453 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:04:38.169599 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:38.169534 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:04:38.169599 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:38.169577 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:04:38.614792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:38.613826 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" event={"ID":"d2715482-f623-43ef-93b8-c92d26626bc9","Type":"ContainerStarted","Data":"9d5b35a1d2a1254aa9c157f1761a9550e9a392da82f4b132db565a6d2c4df49b"} Oct 30 21:04:38.614792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:38.613899 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" event={"ID":"d2715482-f623-43ef-93b8-c92d26626bc9","Type":"ContainerStarted","Data":"c44fd859850cdd96e619729f8dab173480e9304650ecb2f254f986bf07243b9f"} Oct 30 21:04:38.645551 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:38.643652 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" podStartSLOduration=1.643636114 podStartE2EDuration="1.643636114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:38.640047083 +0000 UTC m=+1583.819943616" watchObservedRunningTime="2026-10-30 21:04:38.643636114 +0000 UTC m=+1583.823532620" Oct 30 21:04:38.649918 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:38.649888 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:04:38.751944 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:38.751896 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:39.755270 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:39.755219 4848 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 30 21:04:39.755804 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:39.755417 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" podUID="d2715482-f623-43ef-93b8-c92d26626bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 30 21:04:40.752564 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:40.752523 4848 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 30 21:04:40.752738 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:40.752588 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" podUID="d2715482-f623-43ef-93b8-c92d26626bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 30 21:04:41.761788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:41.757324 4848 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 30 21:04:41.761788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:41.757386 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" podUID="d2715482-f623-43ef-93b8-c92d26626bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 30 21:04:42.755792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:42.753270 4848 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 30 21:04:42.755792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:42.753335 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" podUID="d2715482-f623-43ef-93b8-c92d26626bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 30 21:04:43.634304 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:43.634274 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" Oct 30 21:04:43.752526 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:43.752489 4848 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 30 21:04:43.752680 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:43.752558 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" podUID="d2715482-f623-43ef-93b8-c92d26626bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 30 21:04:44.752516 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:44.752465 4848 patch_prober.go:29] interesting pod/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body= Oct 30 21:04:44.752914 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:44.752618 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" podUID="d2715482-f623-43ef-93b8-c92d26626bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" Oct 30 21:04:45.759375 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:45.759343 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:45.760012 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:45.759721 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:45.760549 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:45.760527 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:04:46.774242 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:46.774207 4848 generic.go:356] "Generic (PLEG): container finished" podID="ebde61df-6d3a-43f2-8ab0-e61c220a5259" containerID="b5a6137eac4ae7e616ed1b7897b1d7a9b81bad785ab7e96e43ee25468eeaf6bf" exitCode=0 Oct 30 21:04:46.774695 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:46.774279 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" event={"ID":"ebde61df-6d3a-43f2-8ab0-e61c220a5259","Type":"ContainerDied","Data":"b5a6137eac4ae7e616ed1b7897b1d7a9b81bad785ab7e96e43ee25468eeaf6bf"} Oct 30 21:04:46.775014 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:46.775000 4848 scope.go:122] "RemoveContainer" containerID="b5a6137eac4ae7e616ed1b7897b1d7a9b81bad785ab7e96e43ee25468eeaf6bf" Oct 30 21:04:46.776591 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:46.776559 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:04:46.776685 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:46.776642 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:04:46.776731 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:46.776689 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:04:47.910147 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:47.910109 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" event={"ID":"ebde61df-6d3a-43f2-8ab0-e61c220a5259","Type":"ContainerStarted","Data":"4210d4095055540cf3dd7f289005f56248921fca8eca50fc7a657243968ab9c5"} Oct 30 21:04:48.758207 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:48.758172 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:04:48.766038 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:48.766000 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 21:04:48.766235 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:48.766103 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" podUID="ebde61df-6d3a-43f2-8ab0-e61c220a5259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 21:04:49.758269 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:49.758232 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 21:04:49.758815 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:49.758288 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" podUID="ebde61df-6d3a-43f2-8ab0-e61c220a5259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 21:04:50.760615 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:50.760145 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 21:04:50.760615 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:50.760221 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" podUID="ebde61df-6d3a-43f2-8ab0-e61c220a5259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 21:04:51.757950 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:51.757912 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 21:04:51.758150 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:51.757978 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" podUID="ebde61df-6d3a-43f2-8ab0-e61c220a5259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 21:04:52.534550 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:04:52.534516 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:04:52.758371 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:52.758336 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 21:04:52.758549 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:52.758396 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" podUID="ebde61df-6d3a-43f2-8ab0-e61c220a5259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 21:04:53.762887 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:53.762855 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:04:53.763302 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:53.763284 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:04:53.764131 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:04:53.764109 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:05:00.589418 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:00.589374 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z"] Oct 30 21:05:00.594075 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:00.594056 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z" Oct 30 21:05:00.601497 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:00.601470 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-49c7c2\"" Oct 30 21:05:00.601645 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:00.601559 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-49c7c2-dockercfg-cmgdm\"" Oct 30 21:05:00.606272 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:00.606246 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z"] Oct 30 21:05:00.690167 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:00.690133 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l8px\" (UniqueName: \"kubernetes.io/projected/8588a492-0418-481f-8a0c-913be7c9b50e-kube-api-access-7l8px\") pod \"payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z\" (UID: \"8588a492-0418-481f-8a0c-913be7c9b50e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z" Oct 30 21:05:00.700514 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:00.690746 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8588a492-0418-481f-8a0c-913be7c9b50e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z\" (UID: \"8588a492-0418-481f-8a0c-913be7c9b50e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z" Oct 30 21:05:00.811878 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:00.811041 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l8px\" (UniqueName: \"kubernetes.io/projected/8588a492-0418-481f-8a0c-913be7c9b50e-kube-api-access-7l8px\") pod \"payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z\" (UID: \"8588a492-0418-481f-8a0c-913be7c9b50e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z" Oct 30 21:05:00.811878 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:00.811283 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8588a492-0418-481f-8a0c-913be7c9b50e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z\" (UID: \"8588a492-0418-481f-8a0c-913be7c9b50e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z" Oct 30 21:05:00.912172 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:00.910618 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l8px\" (UniqueName: \"kubernetes.io/projected/8588a492-0418-481f-8a0c-913be7c9b50e-kube-api-access-7l8px\") pod \"payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z\" (UID: \"8588a492-0418-481f-8a0c-913be7c9b50e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z" Oct 30 21:05:00.968663 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:00.968628 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8588a492-0418-481f-8a0c-913be7c9b50e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z\" (UID: \"8588a492-0418-481f-8a0c-913be7c9b50e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z" Oct 30 21:05:01.221078 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:01.221042 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z" Oct 30 21:05:01.550228 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:01.550201 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z"] Oct 30 21:05:01.551363 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:05:01.551332 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8588a492_0418_481f_8a0c_913be7c9b50e.slice/crio-0ea965524454df931f4796f27d7023fac46646df6e8d75d8049775ff80e17216 WatchSource:0}: Error finding container 0ea965524454df931f4796f27d7023fac46646df6e8d75d8049775ff80e17216: Status 404 returned error can't find the container with id 0ea965524454df931f4796f27d7023fac46646df6e8d75d8049775ff80e17216 Oct 30 21:05:02.266760 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:02.266721 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z" event={"ID":"8588a492-0418-481f-8a0c-913be7c9b50e","Type":"ContainerStarted","Data":"3ca9a2b1aa9eab5a46cee0ce37e49593e422c589b019f7e4abbab9ac48ecae20"} Oct 30 21:05:02.267437 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:02.267410 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z" event={"ID":"8588a492-0418-481f-8a0c-913be7c9b50e","Type":"ContainerStarted","Data":"0ea965524454df931f4796f27d7023fac46646df6e8d75d8049775ff80e17216"} Oct 30 21:05:02.267901 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:02.267880 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z" Oct 30 21:05:02.311692 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:02.311623 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z" podStartSLOduration=2.311599351 podStartE2EDuration="2.311599351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:02.297533062 +0000 UTC m=+1607.477429569" watchObservedRunningTime="2026-10-30 21:05:02.311599351 +0000 UTC m=+1607.491495858" Oct 30 21:05:08.027124 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:08.027079 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h"] Oct 30 21:05:08.030778 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:08.030735 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe65968e-pt79h"] Oct 30 21:05:08.934631 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:08.934602 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cbd5d2a-36e7-4b27-afb0-a94a1465278b" path="/var/lib/kubelet/pods/2cbd5d2a-36e7-4b27-afb0-a94a1465278b/volumes" Oct 30 21:05:09.244991 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:09.244892 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:05:09.244991 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:09.244972 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" podUID="eae7f986-ff45-410c-8301-3e23132e0bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:05:11.388385 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:11.388350 4848 generic.go:356] "Generic (PLEG): container finished" podID="eae7f986-ff45-410c-8301-3e23132e0bc5" containerID="2d70a49c564e94d908b5e3f67166c4cf650bd120f2cb1960410f0e6500def992" exitCode=0 Oct 30 21:05:11.389006 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:11.388434 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" event={"ID":"eae7f986-ff45-410c-8301-3e23132e0bc5","Type":"ContainerDied","Data":"2d70a49c564e94d908b5e3f67166c4cf650bd120f2cb1960410f0e6500def992"} Oct 30 21:05:11.389084 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:11.389050 4848 scope.go:122] "RemoveContainer" containerID="2d70a49c564e94d908b5e3f67166c4cf650bd120f2cb1960410f0e6500def992" Oct 30 21:05:11.391158 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:11.391117 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:05:11.391274 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:11.391214 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:05:11.391274 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:11.391264 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:05:12.155817 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.155779 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7"] Oct 30 21:05:12.160171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.160146 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" Oct 30 21:05:12.193086 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.190808 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7"] Oct 30 21:05:12.193086 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.190983 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-49c7c2-serving-cert\"" Oct 30 21:05:12.236863 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.220869 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7\" (UID: \"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" Oct 30 21:05:12.237194 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.237155 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-metrics-certs\") pod \"maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7\" (UID: \"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" Oct 30 21:05:12.237513 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.237481 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfc2j\" (UniqueName: \"kubernetes.io/projected/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-kube-api-access-tfc2j\") pod \"maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7\" (UID: \"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" Oct 30 21:05:12.237844 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.237818 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7\" (UID: \"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" Oct 30 21:05:12.341793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.339850 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-metrics-certs\") pod \"maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7\" (UID: \"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" Oct 30 21:05:12.341793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.339947 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfc2j\" (UniqueName: \"kubernetes.io/projected/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-kube-api-access-tfc2j\") pod \"maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7\" (UID: \"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" Oct 30 21:05:12.341793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.340042 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7\" (UID: \"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" Oct 30 21:05:12.341793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.340302 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7\" (UID: \"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" Oct 30 21:05:12.341793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.341394 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7\" (UID: \"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" Oct 30 21:05:12.407865 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.401578 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfc2j\" (UniqueName: \"kubernetes.io/projected/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-kube-api-access-tfc2j\") pod \"maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7\" (UID: \"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" Oct 30 21:05:12.420801 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.419173 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-metrics-certs\") pod \"maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7\" (UID: \"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" Oct 30 21:05:12.426800 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.422828 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" event={"ID":"eae7f986-ff45-410c-8301-3e23132e0bc5","Type":"ContainerStarted","Data":"fd67c5c18b6a66d0192249b6f11979b2a03cdb79e667c70b9875ab4a7f616b76"} Oct 30 21:05:12.448457 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.448420 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-maas-api-tls\") pod \"maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7\" (UID: \"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" Oct 30 21:05:12.526805 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.521013 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" Oct 30 21:05:12.538397 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.538361 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:05:12.538641 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.538608 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 21:05:12.538739 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:12.538676 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" podUID="eae7f986-ff45-410c-8301-3e23132e0bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 21:05:13.103354 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:13.103310 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7"] Oct 30 21:05:13.278829 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:13.278804 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z" Oct 30 21:05:13.477775 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:13.477740 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" event={"ID":"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c","Type":"ContainerStarted","Data":"fc918358033edfed4118a9f1efb38cea44a27052b2280fb9fb3b04568473adad"} Oct 30 21:05:13.539662 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:13.539631 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 21:05:13.539851 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:13.539681 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" podUID="eae7f986-ff45-410c-8301-3e23132e0bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 21:05:14.485927 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:14.485893 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" event={"ID":"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c","Type":"ContainerStarted","Data":"fd14d21244cdad9656f8d07168d06be241554a1db77f3395e81f4011b3a70084"} Oct 30 21:05:14.486347 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:14.485953 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" Oct 30 21:05:14.527786 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:14.527712 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" podStartSLOduration=1.976779279 podStartE2EDuration="2.527690495s" totalImagesPullingTime="550.911216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:14.521190973 +0000 UTC m=+1619.701087481" watchObservedRunningTime="2026-10-30 21:05:14.527690495 +0000 UTC m=+1619.707587004" Oct 30 21:05:14.538830 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:14.538800 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 21:05:14.539010 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:14.538850 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" podUID="eae7f986-ff45-410c-8301-3e23132e0bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 21:05:15.538368 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:15.538333 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 21:05:15.538821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:15.538410 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" podUID="eae7f986-ff45-410c-8301-3e23132e0bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 21:05:16.538944 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:16.538889 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 21:05:16.539430 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:16.538975 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" podUID="eae7f986-ff45-410c-8301-3e23132e0bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 21:05:17.089692 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:17.089662 4848 scope.go:122] "RemoveContainer" containerID="40686b69066115ffd9c0019177fdfc0cb5654d8944bd3b7657fba97a5c2d31e5" Oct 30 21:05:17.538371 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:17.538330 4848 patch_prober.go:29] interesting pod/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Oct 30 21:05:17.538564 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:17.538395 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" podUID="eae7f986-ff45-410c-8301-3e23132e0bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Oct 30 21:05:18.543676 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:18.543641 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:05:18.544149 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:18.544119 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:05:18.544853 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:18.544835 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:05:20.498485 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:20.498455 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" Oct 30 21:05:21.035173 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:21.035138 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww"] Oct 30 21:05:21.038562 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:21.038541 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47b45884-56rww"] Oct 30 21:05:22.512581 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:05:22.512542 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:05:22.984849 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:22.984810 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ccfd53-5388-4265-a017-22f7cfb377cb" path="/var/lib/kubelet/pods/21ccfd53-5388-4265-a017-22f7cfb377cb/volumes" Oct 30 21:05:24.026038 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:24.025999 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v"] Oct 30 21:05:24.029361 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:24.029335 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-420b975b-q6c2v"] Oct 30 21:05:24.933655 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:05:24.933620 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c72c5dba-3c91-48e6-90ea-b28ec0a23538" path="/var/lib/kubelet/pods/c72c5dba-3c91-48e6-90ea-b28ec0a23538/volumes" Oct 30 21:05:52.589143 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:05:52.589049 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:06:10.325807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.315950 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27"] Oct 30 21:06:10.349789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.349745 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.359474 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.359433 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27"] Oct 30 21:06:10.362126 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.360002 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-74f9f4-openshift-default-dockercfg-cf7zm\"" Oct 30 21:06:10.414919 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.414886 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-token\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.414919 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.414922 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-credential-socket\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.415188 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.415037 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-podinfo\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.415188 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.415098 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb4dl\" (UniqueName: \"kubernetes.io/projected/dcfcc9ac-8303-4921-b9f8-f7654af44897-kube-api-access-jb4dl\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.415312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.415248 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-workload-socket\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.415377 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.415310 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-workload-certs\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.415441 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.415382 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcfcc9ac-8303-4921-b9f8-f7654af44897-istiod-ca-cert\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.415519 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.415496 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-data\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.415622 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.415604 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-envoy\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.516986 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.516949 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-envoy\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.517181 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.517021 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-token\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.517181 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.517051 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-credential-socket\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.517181 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.517115 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-podinfo\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.517181 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.517149 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb4dl\" (UniqueName: \"kubernetes.io/projected/dcfcc9ac-8303-4921-b9f8-f7654af44897-kube-api-access-jb4dl\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.517420 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.517254 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-workload-socket\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.517420 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.517338 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-workload-certs\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.517420 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.517364 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcfcc9ac-8303-4921-b9f8-f7654af44897-istiod-ca-cert\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.517578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.517440 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-data\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.517578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.517513 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-credential-socket\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.517702 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.517638 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-workload-socket\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.517702 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.517672 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-workload-certs\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.517912 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.517891 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-data\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.518206 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.518181 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcfcc9ac-8303-4921-b9f8-f7654af44897-istiod-ca-cert\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.519247 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.519222 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-envoy\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.519635 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.519612 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-podinfo\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.527267 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.527244 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-token\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.527359 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.527333 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb4dl\" (UniqueName: \"kubernetes.io/projected/dcfcc9ac-8303-4921-b9f8-f7654af44897-kube-api-access-jb4dl\") pod \"e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.686227 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.686199 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:10.973440 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.973412 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27"] Oct 30 21:06:10.975536 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:06:10.975511 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcfcc9ac_8303_4921_b9f8_f7654af44897.slice/crio-237c06e0d6b84a6710a5d42bd569660a3bae9ab538c430ea6b3800002a764696 WatchSource:0}: Error finding container 237c06e0d6b84a6710a5d42bd569660a3bae9ab538c430ea6b3800002a764696: Status 404 returned error can't find the container with id 237c06e0d6b84a6710a5d42bd569660a3bae9ab538c430ea6b3800002a764696 Oct 30 21:06:10.978702 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.978669 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:06:10.978835 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.978746 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:06:10.978915 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:10.978838 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:06:11.065340 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:11.065301 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" event={"ID":"dcfcc9ac-8303-4921-b9f8-f7654af44897","Type":"ContainerStarted","Data":"237c06e0d6b84a6710a5d42bd569660a3bae9ab538c430ea6b3800002a764696"} Oct 30 21:06:12.105791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:12.104047 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" event={"ID":"dcfcc9ac-8303-4921-b9f8-f7654af44897","Type":"ContainerStarted","Data":"79acc039d2cd9635e39a5b6269b2006cdeb360c542eb1c8a5be55a76ba1ff256"} Oct 30 21:06:12.136375 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:12.136297 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" podStartSLOduration=2.136275697 podStartE2EDuration="2.136275697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:12.129182505 +0000 UTC m=+1677.309079013" watchObservedRunningTime="2026-10-30 21:06:12.136275697 +0000 UTC m=+1677.316172204" Oct 30 21:06:12.686507 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:12.686475 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:12.687945 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:12.687920 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 21:06:12.688120 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:12.687969 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" podUID="dcfcc9ac-8303-4921-b9f8-f7654af44897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 21:06:13.702177 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:13.702142 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 21:06:13.702736 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:13.702708 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" podUID="dcfcc9ac-8303-4921-b9f8-f7654af44897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 21:06:14.687501 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:14.687470 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 21:06:14.687685 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:14.687521 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" podUID="dcfcc9ac-8303-4921-b9f8-f7654af44897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 21:06:15.686649 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:15.686611 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 21:06:15.687121 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:15.686682 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" podUID="dcfcc9ac-8303-4921-b9f8-f7654af44897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 21:06:16.706271 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:16.706236 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 21:06:16.706872 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:16.706840 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" podUID="dcfcc9ac-8303-4921-b9f8-f7654af44897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 21:06:17.168500 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:17.168404 4848 scope.go:122] "RemoveContainer" containerID="82830b81f5a97afa4f485c72cc06925007bc05a5ad14c53c5c064ac5108984d7" Oct 30 21:06:17.181670 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:17.181640 4848 scope.go:122] "RemoveContainer" containerID="b32e5159799467af9e3d8392909d58baef0b6ebc86204c6fbd7e8e5c2e7d4030" Oct 30 21:06:17.706793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:17.699745 4848 patch_prober.go:29] interesting pod/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 21:06:17.706793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:17.699848 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" podUID="dcfcc9ac-8303-4921-b9f8-f7654af44897" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 21:06:18.693186 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:18.693153 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:18.693654 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:18.693627 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:18.694296 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:18.694279 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:06:22.511536 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:06:22.511500 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:06:52.559242 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:06:52.559204 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:06:59.629953 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:59.629918 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm"] Oct 30 21:06:59.630594 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:59.630190 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" podUID="9f136686-02a0-4906-a897-9d8a7c1d748d" containerName="istio-proxy" containerID="cri-o://1438d1f7121f547e0bcf4e39b11b7b0bbedf58bd16bb2bcc59af4d64ba7e6df6" gracePeriod=30 Oct 30 21:06:59.633843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:59.633811 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5"] Oct 30 21:06:59.634102 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:59.634059 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" podUID="9835b513-5c93-42f0-90e4-bfc33b2b8a43" containerName="istio-proxy" containerID="cri-o://08c3d64e0a6b713d139b76c403419da5815af0685b9413f463352b7e638b92f1" gracePeriod=30 Oct 30 21:06:59.638516 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:59.638490 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25"] Oct 30 21:06:59.638697 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:06:59.638672 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" podUID="ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" containerName="istio-proxy" containerID="cri-o://dddd23d9dc62bf73bd08590b1ad386ef265d3286438e556da72dd6c5880c4dd5" gracePeriod=30 Oct 30 21:07:04.946466 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:04.946432 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:07:04.982647 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:04.982331 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:07:05.044151 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.044127 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 21:07:05.085582 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.085551 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f136686-02a0-4906-a897-9d8a7c1d748d-kube-api-access-jw2wv\" (UniqueName: \"kubernetes.io/projected/9f136686-02a0-4906-a897-9d8a7c1d748d-kube-api-access-jw2wv\") pod \"9f136686-02a0-4906-a897-9d8a7c1d748d\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " Oct 30 21:07:05.085804 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.085614 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-podinfo\") pod \"9f136686-02a0-4906-a897-9d8a7c1d748d\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " Oct 30 21:07:05.085804 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.085662 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-data\") pod \"9f136686-02a0-4906-a897-9d8a7c1d748d\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " Oct 30 21:07:05.085804 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.085692 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-envoy\") pod \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " Oct 30 21:07:05.085804 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.085719 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-workload-socket\") pod \"9f136686-02a0-4906-a897-9d8a7c1d748d\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " Oct 30 21:07:05.085804 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.085743 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-envoy\") pod \"9f136686-02a0-4906-a897-9d8a7c1d748d\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " Oct 30 21:07:05.086418 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.085813 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istiod-ca-cert\") pod \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " Oct 30 21:07:05.086418 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.085882 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f136686-02a0-4906-a897-9d8a7c1d748d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f136686-02a0-4906-a897-9d8a7c1d748d-istiod-ca-cert\") pod \"9f136686-02a0-4906-a897-9d8a7c1d748d\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " Oct 30 21:07:05.086418 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.085942 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-credential-socket\") pod \"9f136686-02a0-4906-a897-9d8a7c1d748d\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " Oct 30 21:07:05.086418 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.085984 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-token\" (UniqueName: \"kubernetes.io/projected/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-token\") pod \"9f136686-02a0-4906-a897-9d8a7c1d748d\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " Oct 30 21:07:05.086418 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.086035 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9835b513-5c93-42f0-90e4-bfc33b2b8a43-kube-api-access-bvr4c\" (UniqueName: \"kubernetes.io/projected/9835b513-5c93-42f0-90e4-bfc33b2b8a43-kube-api-access-bvr4c\") pod \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " Oct 30 21:07:05.086418 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.086106 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-data\") pod \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " Oct 30 21:07:05.086418 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.086179 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-token\" (UniqueName: \"kubernetes.io/projected/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-token\") pod \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " Oct 30 21:07:05.086418 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.086221 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-credential-socket\") pod \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " Oct 30 21:07:05.086418 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.086366 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-podinfo\") pod \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " Oct 30 21:07:05.087099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.086428 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-workload-certs\") pod \"9f136686-02a0-4906-a897-9d8a7c1d748d\" (UID: \"9f136686-02a0-4906-a897-9d8a7c1d748d\") " Oct 30 21:07:05.087099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.086501 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-workload-certs\") pod \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " Oct 30 21:07:05.087099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.086540 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-workload-socket\") pod \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\" (UID: \"9835b513-5c93-42f0-90e4-bfc33b2b8a43\") " Oct 30 21:07:05.087099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.086938 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-workload-socket" pod "9f136686-02a0-4906-a897-9d8a7c1d748d" (UID: "9f136686-02a0-4906-a897-9d8a7c1d748d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:07:05.087707 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.087678 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.088059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.088032 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f136686-02a0-4906-a897-9d8a7c1d748d-istiod-ca-cert" pod "9f136686-02a0-4906-a897-9d8a7c1d748d" (UID: "9f136686-02a0-4906-a897-9d8a7c1d748d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:07:05.088371 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.088335 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-credential-socket" pod "9f136686-02a0-4906-a897-9d8a7c1d748d" (UID: "9f136686-02a0-4906-a897-9d8a7c1d748d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:07:05.088506 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.088485 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-workload-socket" pod "9835b513-5c93-42f0-90e4-bfc33b2b8a43" (UID: "9835b513-5c93-42f0-90e4-bfc33b2b8a43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:07:05.089187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.089159 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-workload-certs" pod "9835b513-5c93-42f0-90e4-bfc33b2b8a43" (UID: "9835b513-5c93-42f0-90e4-bfc33b2b8a43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:07:05.089718 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.089691 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istiod-ca-cert" pod "9835b513-5c93-42f0-90e4-bfc33b2b8a43" (UID: "9835b513-5c93-42f0-90e4-bfc33b2b8a43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:07:05.090022 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.089998 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-data" pod "9f136686-02a0-4906-a897-9d8a7c1d748d" (UID: "9f136686-02a0-4906-a897-9d8a7c1d748d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:07:05.090199 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.090178 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-credential-socket" pod "9835b513-5c93-42f0-90e4-bfc33b2b8a43" (UID: "9835b513-5c93-42f0-90e4-bfc33b2b8a43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:07:05.090620 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.090469 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f136686-02a0-4906-a897-9d8a7c1d748d-kube-api-access-jw2wv" pod "9f136686-02a0-4906-a897-9d8a7c1d748d" (UID: "9f136686-02a0-4906-a897-9d8a7c1d748d"). InnerVolumeSpecName "kube-api-access-jw2wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:07:05.090736 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.090716 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-data" pod "9835b513-5c93-42f0-90e4-bfc33b2b8a43" (UID: "9835b513-5c93-42f0-90e4-bfc33b2b8a43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:07:05.090911 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.090895 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-workload-certs" pod "9f136686-02a0-4906-a897-9d8a7c1d748d" (UID: "9f136686-02a0-4906-a897-9d8a7c1d748d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:07:05.091333 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.091311 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-podinfo" pod "9f136686-02a0-4906-a897-9d8a7c1d748d" (UID: "9f136686-02a0-4906-a897-9d8a7c1d748d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:07:05.091417 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.091376 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-envoy" pod "9f136686-02a0-4906-a897-9d8a7c1d748d" (UID: "9f136686-02a0-4906-a897-9d8a7c1d748d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:07:05.091514 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.091498 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-envoy" pod "9835b513-5c93-42f0-90e4-bfc33b2b8a43" (UID: "9835b513-5c93-42f0-90e4-bfc33b2b8a43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:07:05.093171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.093146 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-token" pod "9f136686-02a0-4906-a897-9d8a7c1d748d" (UID: "9f136686-02a0-4906-a897-9d8a7c1d748d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:07:05.093171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.093148 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-podinfo" pod "9835b513-5c93-42f0-90e4-bfc33b2b8a43" (UID: "9835b513-5c93-42f0-90e4-bfc33b2b8a43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:07:05.093338 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.093183 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-token" pod "9835b513-5c93-42f0-90e4-bfc33b2b8a43" (UID: "9835b513-5c93-42f0-90e4-bfc33b2b8a43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:07:05.094625 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.094597 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9835b513-5c93-42f0-90e4-bfc33b2b8a43-kube-api-access-bvr4c" pod "9835b513-5c93-42f0-90e4-bfc33b2b8a43" (UID: "9835b513-5c93-42f0-90e4-bfc33b2b8a43"). InnerVolumeSpecName "kube-api-access-bvr4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:07:05.188182 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.188149 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-credential-socket\") pod \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " Oct 30 21:07:05.188408 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.188255 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-podinfo\") pod \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " Oct 30 21:07:05.188408 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.188343 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-data\") pod \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " Oct 30 21:07:05.188408 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.188365 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-workload-socket\") pod \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " Oct 30 21:07:05.188597 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.188456 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-credential-socket" pod "ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" (UID: "ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:07:05.188597 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.188539 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-workload-certs\") pod \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " Oct 30 21:07:05.188597 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.188572 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-envoy\") pod \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " Oct 30 21:07:05.188809 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.188608 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-kube-api-access-n4z2s\" (UniqueName: \"kubernetes.io/projected/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-kube-api-access-n4z2s\") pod \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " Oct 30 21:07:05.188809 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.188626 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-data" pod "ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" (UID: "ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:07:05.188809 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.188640 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-workload-socket" pod "ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" (UID: "ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:07:05.188809 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.188671 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-token\" (UniqueName: \"kubernetes.io/projected/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-token\") pod \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " Oct 30 21:07:05.188809 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.188710 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istiod-ca-cert\") pod \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\" (UID: \"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2\") " Oct 30 21:07:05.189083 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.188849 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-workload-certs" pod "ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" (UID: "ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:07:05.189163 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.189131 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istiod-ca-cert" pod "ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" (UID: "ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:07:05.189443 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.189416 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.189578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.189448 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.189578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.189465 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvr4c\" (UniqueName: \"kubernetes.io/projected/9835b513-5c93-42f0-90e4-bfc33b2b8a43-kube-api-access-bvr4c\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.189578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.189482 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.189578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.189496 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.189578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.189511 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.189578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.189525 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.189578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.189542 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.189578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.189556 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.190118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.189570 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.190118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.189975 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.190118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.189992 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.190118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.190006 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.190118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.190021 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw2wv\" (UniqueName: \"kubernetes.io/projected/9f136686-02a0-4906-a897-9d8a7c1d748d-kube-api-access-jw2wv\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.190118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.190036 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.190118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.190049 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.190118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.190065 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.190118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.190078 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f136686-02a0-4906-a897-9d8a7c1d748d-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.190118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.190091 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9835b513-5c93-42f0-90e4-bfc33b2b8a43-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.190118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.190104 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.190118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.190117 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.190619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.190131 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f136686-02a0-4906-a897-9d8a7c1d748d-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.190943 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.190920 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-envoy" pod "ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" (UID: "ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:07:05.191159 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.191141 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-podinfo" pod "ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" (UID: "ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:07:05.191237 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.191229 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-token" pod "ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" (UID: "ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:07:05.191342 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.191275 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-kube-api-access-n4z2s" pod "ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" (UID: "ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2"). InnerVolumeSpecName "kube-api-access-n4z2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:07:05.298660 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.298529 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.298660 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.298577 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4z2s\" (UniqueName: \"kubernetes.io/projected/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-kube-api-access-n4z2s\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.298660 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.298592 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.298660 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.298606 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:05.744273 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.744237 4848 generic.go:356] "Generic (PLEG): container finished" podID="ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" containerID="dddd23d9dc62bf73bd08590b1ad386ef265d3286438e556da72dd6c5880c4dd5" exitCode=0 Oct 30 21:07:05.744481 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.744297 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" Oct 30 21:07:05.744481 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.744332 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" event={"ID":"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2","Type":"ContainerDied","Data":"dddd23d9dc62bf73bd08590b1ad386ef265d3286438e556da72dd6c5880c4dd5"} Oct 30 21:07:05.744481 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.744416 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25" event={"ID":"ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2","Type":"ContainerDied","Data":"68a17bc2b3f4aaef72eee5adc30b8ccbb28b83b42a48980be3c62776fab823c3"} Oct 30 21:07:05.744481 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.744448 4848 scope.go:122] "RemoveContainer" containerID="dddd23d9dc62bf73bd08590b1ad386ef265d3286438e556da72dd6c5880c4dd5" Oct 30 21:07:05.745808 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.745748 4848 generic.go:356] "Generic (PLEG): container finished" podID="9f136686-02a0-4906-a897-9d8a7c1d748d" containerID="1438d1f7121f547e0bcf4e39b11b7b0bbedf58bd16bb2bcc59af4d64ba7e6df6" exitCode=0 Oct 30 21:07:05.745904 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.745808 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" event={"ID":"9f136686-02a0-4906-a897-9d8a7c1d748d","Type":"ContainerDied","Data":"1438d1f7121f547e0bcf4e39b11b7b0bbedf58bd16bb2bcc59af4d64ba7e6df6"} Oct 30 21:07:05.745904 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.745863 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" event={"ID":"9f136686-02a0-4906-a897-9d8a7c1d748d","Type":"ContainerDied","Data":"65132b009c5e8d1eb12d91ea5fdf6510ae30234c225ce0195ac91543ece5c4d7"} Oct 30 21:07:05.745904 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.745822 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm" Oct 30 21:07:05.758092 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.757684 4848 generic.go:356] "Generic (PLEG): container finished" podID="9835b513-5c93-42f0-90e4-bfc33b2b8a43" containerID="08c3d64e0a6b713d139b76c403419da5815af0685b9413f463352b7e638b92f1" exitCode=0 Oct 30 21:07:05.758092 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.757755 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" Oct 30 21:07:05.758092 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.758014 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" event={"ID":"9835b513-5c93-42f0-90e4-bfc33b2b8a43","Type":"ContainerDied","Data":"08c3d64e0a6b713d139b76c403419da5815af0685b9413f463352b7e638b92f1"} Oct 30 21:07:05.758092 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.758065 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5" event={"ID":"9835b513-5c93-42f0-90e4-bfc33b2b8a43","Type":"ContainerDied","Data":"6a7178261bc063bcf7e9cee0e2d982b90aeb837f040c466a84110bed27ec4b69"} Oct 30 21:07:05.758828 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.758760 4848 scope.go:122] "RemoveContainer" containerID="dddd23d9dc62bf73bd08590b1ad386ef265d3286438e556da72dd6c5880c4dd5" Oct 30 21:07:05.762644 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:07:05.762609 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dddd23d9dc62bf73bd08590b1ad386ef265d3286438e556da72dd6c5880c4dd5\": container with ID starting with dddd23d9dc62bf73bd08590b1ad386ef265d3286438e556da72dd6c5880c4dd5 not found: ID does not exist" containerID="dddd23d9dc62bf73bd08590b1ad386ef265d3286438e556da72dd6c5880c4dd5" Oct 30 21:07:05.762821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.762649 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dddd23d9dc62bf73bd08590b1ad386ef265d3286438e556da72dd6c5880c4dd5"} err="failed to get container status \"dddd23d9dc62bf73bd08590b1ad386ef265d3286438e556da72dd6c5880c4dd5\": rpc error: code = NotFound desc = could not find container \"dddd23d9dc62bf73bd08590b1ad386ef265d3286438e556da72dd6c5880c4dd5\": container with ID starting with dddd23d9dc62bf73bd08590b1ad386ef265d3286438e556da72dd6c5880c4dd5 not found: ID does not exist" Oct 30 21:07:05.762821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.762674 4848 scope.go:122] "RemoveContainer" containerID="1438d1f7121f547e0bcf4e39b11b7b0bbedf58bd16bb2bcc59af4d64ba7e6df6" Oct 30 21:07:05.775077 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.775054 4848 scope.go:122] "RemoveContainer" containerID="1438d1f7121f547e0bcf4e39b11b7b0bbedf58bd16bb2bcc59af4d64ba7e6df6" Oct 30 21:07:05.775403 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:07:05.775386 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1438d1f7121f547e0bcf4e39b11b7b0bbedf58bd16bb2bcc59af4d64ba7e6df6\": container with ID starting with 1438d1f7121f547e0bcf4e39b11b7b0bbedf58bd16bb2bcc59af4d64ba7e6df6 not found: ID does not exist" containerID="1438d1f7121f547e0bcf4e39b11b7b0bbedf58bd16bb2bcc59af4d64ba7e6df6" Oct 30 21:07:05.775462 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.775410 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1438d1f7121f547e0bcf4e39b11b7b0bbedf58bd16bb2bcc59af4d64ba7e6df6"} err="failed to get container status \"1438d1f7121f547e0bcf4e39b11b7b0bbedf58bd16bb2bcc59af4d64ba7e6df6\": rpc error: code = NotFound desc = could not find container \"1438d1f7121f547e0bcf4e39b11b7b0bbedf58bd16bb2bcc59af4d64ba7e6df6\": container with ID starting with 1438d1f7121f547e0bcf4e39b11b7b0bbedf58bd16bb2bcc59af4d64ba7e6df6 not found: ID does not exist" Oct 30 21:07:05.775462 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.775427 4848 scope.go:122] "RemoveContainer" containerID="08c3d64e0a6b713d139b76c403419da5815af0685b9413f463352b7e638b92f1" Oct 30 21:07:05.785553 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.785532 4848 scope.go:122] "RemoveContainer" containerID="08c3d64e0a6b713d139b76c403419da5815af0685b9413f463352b7e638b92f1" Oct 30 21:07:05.786031 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:07:05.785873 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08c3d64e0a6b713d139b76c403419da5815af0685b9413f463352b7e638b92f1\": container with ID starting with 08c3d64e0a6b713d139b76c403419da5815af0685b9413f463352b7e638b92f1 not found: ID does not exist" containerID="08c3d64e0a6b713d139b76c403419da5815af0685b9413f463352b7e638b92f1" Oct 30 21:07:05.786292 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.786047 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c3d64e0a6b713d139b76c403419da5815af0685b9413f463352b7e638b92f1"} err="failed to get container status \"08c3d64e0a6b713d139b76c403419da5815af0685b9413f463352b7e638b92f1\": rpc error: code = NotFound desc = could not find container \"08c3d64e0a6b713d139b76c403419da5815af0685b9413f463352b7e638b92f1\": container with ID starting with 08c3d64e0a6b713d139b76c403419da5815af0685b9413f463352b7e638b92f1 not found: ID does not exist" Oct 30 21:07:05.787418 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.787395 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm"] Oct 30 21:07:05.791150 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.791129 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-fmxnm"] Oct 30 21:07:05.801033 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.801007 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25"] Oct 30 21:07:05.806146 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.806124 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-nmq25"] Oct 30 21:07:05.824071 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.824045 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5"] Oct 30 21:07:05.839871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:05.839846 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-xmsg5"] Oct 30 21:07:06.974967 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:06.974925 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9835b513-5c93-42f0-90e4-bfc33b2b8a43" path="/var/lib/kubelet/pods/9835b513-5c93-42f0-90e4-bfc33b2b8a43/volumes" Oct 30 21:07:06.976381 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:06.976355 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f136686-02a0-4906-a897-9d8a7c1d748d" path="/var/lib/kubelet/pods/9f136686-02a0-4906-a897-9d8a7c1d748d/volumes" Oct 30 21:07:06.977366 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:06.977344 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" path="/var/lib/kubelet/pods/ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2/volumes" Oct 30 21:07:11.575126 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.575088 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x"] Oct 30 21:07:11.576725 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.576696 4848 cpu_manager.go:414] "containerMap: removing container" podUID="ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" containerName="istio-proxy" Oct 30 21:07:11.576725 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.576727 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" containerName="istio-proxy" Oct 30 21:07:11.576901 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.576778 4848 cpu_manager.go:414] "containerMap: removing container" podUID="9835b513-5c93-42f0-90e4-bfc33b2b8a43" containerName="istio-proxy" Oct 30 21:07:11.576901 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.576788 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9835b513-5c93-42f0-90e4-bfc33b2b8a43" containerName="istio-proxy" Oct 30 21:07:11.576901 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.576853 4848 cpu_manager.go:414] "containerMap: removing container" podUID="9f136686-02a0-4906-a897-9d8a7c1d748d" containerName="istio-proxy" Oct 30 21:07:11.576901 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.576862 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f136686-02a0-4906-a897-9d8a7c1d748d" containerName="istio-proxy" Oct 30 21:07:11.577066 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.577052 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2c7b5c-94f9-45b8-ad43-de9a1f5110f2" containerName="istio-proxy" Oct 30 21:07:11.577099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.577073 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="9835b513-5c93-42f0-90e4-bfc33b2b8a43" containerName="istio-proxy" Oct 30 21:07:11.577099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.577093 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f136686-02a0-4906-a897-9d8a7c1d748d" containerName="istio-proxy" Oct 30 21:07:11.583441 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.583418 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.596613 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.596582 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x"] Oct 30 21:07:11.596756 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.596628 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-a574c2-openshift-default-dockercfg-pm5sv\"" Oct 30 21:07:11.641444 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.641403 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d85436-702d-4fb3-8e98-220cb26ef773-istio-token\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.641627 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.641565 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-workload-certs\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.641627 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.641595 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-istio-data\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.641627 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.641621 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-workload-socket\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.641814 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.641791 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-credential-socket\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.641853 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.641839 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d85436-702d-4fb3-8e98-220cb26ef773-istio-podinfo\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.641899 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.641888 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d85436-702d-4fb3-8e98-220cb26ef773-istiod-ca-cert\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.641999 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.641981 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vwr4\" (UniqueName: \"kubernetes.io/projected/05d85436-702d-4fb3-8e98-220cb26ef773-kube-api-access-8vwr4\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.642121 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.642103 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-istio-envoy\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.773246 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.757261 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-workload-certs\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.773246 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.757312 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-istio-data\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.773246 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.757348 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-workload-socket\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.773246 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.757446 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-credential-socket\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.773246 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.757483 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d85436-702d-4fb3-8e98-220cb26ef773-istio-podinfo\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.773246 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.757563 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d85436-702d-4fb3-8e98-220cb26ef773-istiod-ca-cert\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.773246 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.757614 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vwr4\" (UniqueName: \"kubernetes.io/projected/05d85436-702d-4fb3-8e98-220cb26ef773-kube-api-access-8vwr4\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.773246 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.757786 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-istio-envoy\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.773246 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.757819 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d85436-702d-4fb3-8e98-220cb26ef773-istio-token\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.773246 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.758559 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-workload-certs\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.773246 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.758855 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-istio-data\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.773246 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.759141 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-workload-socket\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.773246 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.759373 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-credential-socket\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.794800 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.791054 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d85436-702d-4fb3-8e98-220cb26ef773-istiod-ca-cert\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.883283 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.883156 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-istio-envoy\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.886822 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.884099 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d85436-702d-4fb3-8e98-220cb26ef773-istio-podinfo\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:11.972891 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:11.964737 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d85436-702d-4fb3-8e98-220cb26ef773-istio-token\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:12.006803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:12.003957 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vwr4\" (UniqueName: \"kubernetes.io/projected/05d85436-702d-4fb3-8e98-220cb26ef773-kube-api-access-8vwr4\") pod \"e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:12.201235 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:12.201205 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:12.486174 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:12.486148 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x"] Oct 30 21:07:12.487138 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:07:12.487112 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d85436_702d_4fb3_8e98_220cb26ef773.slice/crio-3987d5eda915e483d5acb4b75a81edb677219a6ba54bbf535551653f2d78b856 WatchSource:0}: Error finding container 3987d5eda915e483d5acb4b75a81edb677219a6ba54bbf535551653f2d78b856: Status 404 returned error can't find the container with id 3987d5eda915e483d5acb4b75a81edb677219a6ba54bbf535551653f2d78b856 Oct 30 21:07:12.490608 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:12.490564 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:07:12.490715 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:12.490657 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:07:12.490715 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:12.490703 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:07:12.945403 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:12.945365 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" event={"ID":"05d85436-702d-4fb3-8e98-220cb26ef773","Type":"ContainerStarted","Data":"353971ad64b4eee065b9e6a70897017bc4cd15f08639b81452e92b2a7de0d4a1"} Oct 30 21:07:12.945831 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:12.945430 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" event={"ID":"05d85436-702d-4fb3-8e98-220cb26ef773","Type":"ContainerStarted","Data":"3987d5eda915e483d5acb4b75a81edb677219a6ba54bbf535551653f2d78b856"} Oct 30 21:07:12.964282 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:12.964215 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" podStartSLOduration=1.964202903 podStartE2EDuration="1.964202903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:12.962988165 +0000 UTC m=+1738.142884671" watchObservedRunningTime="2026-10-30 21:07:12.964202903 +0000 UTC m=+1738.144099410" Oct 30 21:07:13.202284 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:13.201874 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:13.217904 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:13.217860 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:13.952715 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:13.952684 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:13.953710 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:13.953693 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:07:16.455132 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:16.455090 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg"] Oct 30 21:07:16.460103 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:16.460081 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg" Oct 30 21:07:16.469867 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:16.469841 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg"] Oct 30 21:07:16.471017 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:16.470985 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-a574c2\"" Oct 30 21:07:16.472003 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:16.471980 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-a574c2-dockercfg-7n895\"" Oct 30 21:07:16.626797 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:16.616048 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg\" (UID: \"a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg" Oct 30 21:07:16.626797 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:16.616219 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbqcx\" (UniqueName: \"kubernetes.io/projected/a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6-kube-api-access-tbqcx\") pod \"payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg\" (UID: \"a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg" Oct 30 21:07:16.738884 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:16.730021 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg\" (UID: \"a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg" Oct 30 21:07:16.738884 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:16.730282 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbqcx\" (UniqueName: \"kubernetes.io/projected/a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6-kube-api-access-tbqcx\") pod \"payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg\" (UID: \"a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg" Oct 30 21:07:16.738884 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:16.731931 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg\" (UID: \"a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg" Oct 30 21:07:16.784790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:16.784239 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbqcx\" (UniqueName: \"kubernetes.io/projected/a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6-kube-api-access-tbqcx\") pod \"payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg\" (UID: \"a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg" Oct 30 21:07:17.081323 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:17.080981 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg" Oct 30 21:07:17.254663 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:17.254634 4848 scope.go:122] "RemoveContainer" containerID="40d2459d72c7ff4a8f2bc73651b8ab47a47d5178729400cf653a58ec73335ab2" Oct 30 21:07:17.387979 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:17.387948 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg"] Oct 30 21:07:17.388542 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:07:17.388508 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29d6d7f_34e8_4bfa_aabe_15e84c94c8f6.slice/crio-b4533d59c34a8b601ab2696a5bba6711ca5bdda07f95dda946e741e4b0cb8bd1 WatchSource:0}: Error finding container b4533d59c34a8b601ab2696a5bba6711ca5bdda07f95dda946e741e4b0cb8bd1: Status 404 returned error can't find the container with id b4533d59c34a8b601ab2696a5bba6711ca5bdda07f95dda946e741e4b0cb8bd1 Oct 30 21:07:17.981523 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:17.981482 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg" event={"ID":"a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6","Type":"ContainerStarted","Data":"ac0c55433391c09afe73ca33f5344ba831cbced5cde43f74d475d0e65ade201b"} Oct 30 21:07:17.981945 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:17.981545 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg" event={"ID":"a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6","Type":"ContainerStarted","Data":"b4533d59c34a8b601ab2696a5bba6711ca5bdda07f95dda946e741e4b0cb8bd1"} Oct 30 21:07:17.981945 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:17.981658 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg" Oct 30 21:07:18.002421 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:18.002373 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg" podStartSLOduration=2.002359651 podStartE2EDuration="2.002359651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:17.998821467 +0000 UTC m=+1743.178717978" watchObservedRunningTime="2026-10-30 21:07:18.002359651 +0000 UTC m=+1743.182256157" Oct 30 21:07:19.192136 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:19.192101 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72"] Oct 30 21:07:19.206394 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:19.206367 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc"] Oct 30 21:07:22.513888 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:07:22.513850 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:07:24.192602 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:24.192557 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72" podUID="fd1bad1a-2f48-4b2e-801d-ad3762be02c8" containerName="payload-processing" containerID="cri-o://32cd53445921a9c54c082566c8a2a58114f3e457a82c649381c5766a9b057513" gracePeriod=25 Oct 30 21:07:24.207120 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:24.207086 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc" podUID="711bf26e-5e4f-4b03-93c2-5302e22b543d" containerName="payload-pre-processing" containerID="cri-o://3212d060a18b1c6e5606169a0485e2f8a6660349c3e3cf2e6557effdbaaf73dc" gracePeriod=25 Oct 30 21:07:27.241261 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:27.241220 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc" podUID="711bf26e-5e4f-4b03-93c2-5302e22b543d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:07:27.241687 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:27.241460 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72" podUID="fd1bad1a-2f48-4b2e-801d-ad3762be02c8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:07:28.304094 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:28.304056 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw"] Oct 30 21:07:28.308922 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:28.308896 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" Oct 30 21:07:28.323774 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:28.323719 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw"] Oct 30 21:07:28.332117 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:28.332088 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-a574c2-serving-cert\"" Oct 30 21:07:28.479402 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:28.479372 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw\" (UID: \"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" Oct 30 21:07:28.479578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:28.479412 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw\" (UID: \"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" Oct 30 21:07:28.479630 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:28.479606 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-metrics-certs\") pod \"maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw\" (UID: \"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" Oct 30 21:07:28.479784 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:28.479757 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g29n\" (UniqueName: \"kubernetes.io/projected/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-kube-api-access-7g29n\") pod \"maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw\" (UID: \"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" Oct 30 21:07:28.582840 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:28.580977 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw\" (UID: \"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" Oct 30 21:07:28.582840 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:28.581144 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-metrics-certs\") pod \"maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw\" (UID: \"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" Oct 30 21:07:28.582840 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:28.581347 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g29n\" (UniqueName: \"kubernetes.io/projected/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-kube-api-access-7g29n\") pod \"maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw\" (UID: \"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" Oct 30 21:07:28.582840 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:28.581464 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw\" (UID: \"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" Oct 30 21:07:28.592782 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:28.592722 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw\" (UID: \"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" Oct 30 21:07:28.633273 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:28.632825 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-metrics-certs\") pod \"maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw\" (UID: \"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" Oct 30 21:07:28.645247 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:28.645214 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g29n\" (UniqueName: \"kubernetes.io/projected/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-kube-api-access-7g29n\") pod \"maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw\" (UID: \"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" Oct 30 21:07:28.646132 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:28.646106 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw\" (UID: \"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" Oct 30 21:07:28.938680 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:28.938645 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" Oct 30 21:07:29.060789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:29.046977 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg" Oct 30 21:07:29.415682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:29.415650 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw"] Oct 30 21:07:29.416005 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:07:29.415973 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83cf9cb4_12c1_47dc_8e95_d1ca0ee6ea18.slice/crio-819a7f9e48f70aac41be9374002c199b37324df3eae37846b9168c86cd6d2ac3 WatchSource:0}: Error finding container 819a7f9e48f70aac41be9374002c199b37324df3eae37846b9168c86cd6d2ac3: Status 404 returned error can't find the container with id 819a7f9e48f70aac41be9374002c199b37324df3eae37846b9168c86cd6d2ac3 Oct 30 21:07:30.197348 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:30.197308 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" event={"ID":"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18","Type":"ContainerStarted","Data":"e2aa53b437beb9b90331b5aec4e696f56455d98dce7f9c1078909173d0f9c8b2"} Oct 30 21:07:30.197961 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:30.197390 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" Oct 30 21:07:30.197961 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:30.197414 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" event={"ID":"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18","Type":"ContainerStarted","Data":"819a7f9e48f70aac41be9374002c199b37324df3eae37846b9168c86cd6d2ac3"} Oct 30 21:07:30.216709 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:30.216656 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" podStartSLOduration=1.810413095 podStartE2EDuration="2.216639909s" totalImagesPullingTime="406.226814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:30.216399859 +0000 UTC m=+1755.396296366" watchObservedRunningTime="2026-10-30 21:07:30.216639909 +0000 UTC m=+1755.396536415" Oct 30 21:07:36.138626 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.138594 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c"] Oct 30 21:07:36.143654 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.143632 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.148915 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.148890 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-6538cc-openshift-default-dockercfg-qpjhq\"" Oct 30 21:07:36.160264 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.160235 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c"] Oct 30 21:07:36.220406 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.220368 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" Oct 30 21:07:36.221267 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.221233 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-podinfo\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.221401 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.221319 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-workload-socket\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.221479 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.221410 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-credential-socket\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.221479 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.221469 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kjrz\" (UniqueName: \"kubernetes.io/projected/c8b6880d-49f5-4900-8bfb-6d7332195be4-kube-api-access-6kjrz\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.221598 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.221519 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-data\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.221598 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.221581 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b6880d-49f5-4900-8bfb-6d7332195be4-istiod-ca-cert\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.221721 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.221659 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-token\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.221807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.221758 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-envoy\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.221807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.221797 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-workload-certs\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.341099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.339418 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-podinfo\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.341099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.339503 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-workload-socket\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.341099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.339581 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-credential-socket\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.341099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.339640 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kjrz\" (UniqueName: \"kubernetes.io/projected/c8b6880d-49f5-4900-8bfb-6d7332195be4-kube-api-access-6kjrz\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.341099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.339717 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-data\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.341099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.339760 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b6880d-49f5-4900-8bfb-6d7332195be4-istiod-ca-cert\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.341099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.339862 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-token\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.341099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.339975 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-envoy\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.341099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.340016 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-workload-certs\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.341099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.340617 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-workload-certs\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.415698 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.415615 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-data\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.582837 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.582089 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-credential-socket\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.582837 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.582448 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-workload-socket\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.648935 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.648896 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kjrz\" (UniqueName: \"kubernetes.io/projected/c8b6880d-49f5-4900-8bfb-6d7332195be4-kube-api-access-6kjrz\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.664937 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.653419 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b6880d-49f5-4900-8bfb-6d7332195be4-istiod-ca-cert\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.688742 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.688579 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-envoy\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.689429 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.689342 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-podinfo\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.689874 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.689854 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-token\") pod \"e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:36.763154 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:36.763122 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:37.051620 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:07:37.051587 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b6880d_49f5_4900_8bfb_6d7332195be4.slice/crio-7d1b83499838dde9174a01134b60f9080262eac3ea753c655c9901b8d27b7805 WatchSource:0}: Error finding container 7d1b83499838dde9174a01134b60f9080262eac3ea753c655c9901b8d27b7805: Status 404 returned error can't find the container with id 7d1b83499838dde9174a01134b60f9080262eac3ea753c655c9901b8d27b7805 Oct 30 21:07:37.052022 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:37.051998 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c"] Oct 30 21:07:37.054627 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:37.054597 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:07:37.054695 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:37.054662 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:07:37.054737 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:37.054693 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:07:37.254072 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:37.254018 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72" podUID="fd1bad1a-2f48-4b2e-801d-ad3762be02c8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:07:37.255148 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:37.255121 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc" podUID="711bf26e-5e4f-4b03-93c2-5302e22b543d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:07:37.620269 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:37.620233 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" event={"ID":"c8b6880d-49f5-4900-8bfb-6d7332195be4","Type":"ContainerStarted","Data":"486ca2cf193f5a0fbacdb3d1260439c213c223c2770ffaf2340ec3d859d10c87"} Oct 30 21:07:37.620465 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:37.620288 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" event={"ID":"c8b6880d-49f5-4900-8bfb-6d7332195be4","Type":"ContainerStarted","Data":"7d1b83499838dde9174a01134b60f9080262eac3ea753c655c9901b8d27b7805"} Oct 30 21:07:37.641983 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:37.641932 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" podStartSLOduration=1.6419183080000002 podStartE2EDuration="1.641918308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:37.641698016 +0000 UTC m=+1762.821594524" watchObservedRunningTime="2026-10-30 21:07:37.641918308 +0000 UTC m=+1762.821814815" Oct 30 21:07:37.782849 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:37.775887 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:38.781937 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:38.781875 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:39.759797 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:39.757261 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:39.771731 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:39.771418 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:40.267829 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.263391 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45"] Oct 30 21:07:40.267829 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.263651 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" podUID="94768319-93cc-4cc4-ae81-ab6568a965c4" containerName="maas-api" containerID="cri-o://5f5d8a64c68300654efd39a789829c13a8b520e5814bd99cb886680676511662" gracePeriod=30 Oct 30 21:07:40.460952 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.459325 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd"] Oct 30 21:07:40.475107 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.475079 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:40.489665 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.486673 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd"] Oct 30 21:07:40.491011 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.490612 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-envoy\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:40.491011 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.490667 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-795m2\" (UniqueName: \"kubernetes.io/projected/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-kube-api-access-795m2\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:40.491011 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.490709 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-token\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:40.491011 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.490743 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-workload-socket\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:40.491011 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.490924 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-workload-certs\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:40.491011 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.490997 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-credential-socket\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:40.493855 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.491063 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-data\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:40.493855 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.491141 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-podinfo\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:40.493855 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.491217 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istiod-ca-cert\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:40.639072 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.638639 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-99668e-openshift-default-dockercfg-442m8\"" Oct 30 21:07:40.859414 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.859318 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-podinfo\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:40.859746 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.859724 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istiod-ca-cert\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:40.872787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.859976 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-envoy\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:40.873144 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.873114 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-795m2\" (UniqueName: \"kubernetes.io/projected/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-kube-api-access-795m2\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:40.873311 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.873295 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-token\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:40.873427 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.873412 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-workload-socket\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:40.873655 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.873640 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-workload-certs\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:40.873844 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.873829 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-credential-socket\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:40.874023 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.874010 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-data\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:40.951908 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:40.936352 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-workload-socket\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:41.037529 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.037310 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-podinfo\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:41.049116 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.049068 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-workload-certs\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:41.050619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.050584 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-data\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:41.069889 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.049965 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-credential-socket\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:41.088099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.087832 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-envoy\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:41.088099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.087910 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istiod-ca-cert\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:41.101678 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.101643 4848 generic.go:356] "Generic (PLEG): container finished" podID="94768319-93cc-4cc4-ae81-ab6568a965c4" containerID="5f5d8a64c68300654efd39a789829c13a8b520e5814bd99cb886680676511662" exitCode=0 Oct 30 21:07:41.102660 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.102628 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" event={"ID":"94768319-93cc-4cc4-ae81-ab6568a965c4","Type":"ContainerDied","Data":"5f5d8a64c68300654efd39a789829c13a8b520e5814bd99cb886680676511662"} Oct 30 21:07:41.116722 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.115277 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-795m2\" (UniqueName: \"kubernetes.io/projected/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-kube-api-access-795m2\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:41.126497 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.126440 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-token\") pod \"e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:41.159396 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:07:41.159330 4848 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94768319_93cc_4cc4_ae81_ab6568a965c4.slice/crio-5f5d8a64c68300654efd39a789829c13a8b520e5814bd99cb886680676511662.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94768319_93cc_4cc4_ae81_ab6568a965c4.slice/crio-conmon-5f5d8a64c68300654efd39a789829c13a8b520e5814bd99cb886680676511662.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:07:41.378467 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.378396 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" Oct 30 21:07:41.401948 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.399507 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:41.497926 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.497304 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94768319-93cc-4cc4-ae81-ab6568a965c4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94768319-93cc-4cc4-ae81-ab6568a965c4-openshift-service-ca\") pod \"94768319-93cc-4cc4-ae81-ab6568a965c4\" (UID: \"94768319-93cc-4cc4-ae81-ab6568a965c4\") " Oct 30 21:07:41.497926 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.497418 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94768319-93cc-4cc4-ae81-ab6568a965c4-kube-api-access-cvx6p\" (UniqueName: \"kubernetes.io/projected/94768319-93cc-4cc4-ae81-ab6568a965c4-kube-api-access-cvx6p\") pod \"94768319-93cc-4cc4-ae81-ab6568a965c4\" (UID: \"94768319-93cc-4cc4-ae81-ab6568a965c4\") " Oct 30 21:07:41.497926 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.497483 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94768319-93cc-4cc4-ae81-ab6568a965c4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94768319-93cc-4cc4-ae81-ab6568a965c4-maas-api-tls\") pod \"94768319-93cc-4cc4-ae81-ab6568a965c4\" (UID: \"94768319-93cc-4cc4-ae81-ab6568a965c4\") " Oct 30 21:07:41.497926 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.497603 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94768319-93cc-4cc4-ae81-ab6568a965c4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/94768319-93cc-4cc4-ae81-ab6568a965c4-metrics-certs\") pod \"94768319-93cc-4cc4-ae81-ab6568a965c4\" (UID: \"94768319-93cc-4cc4-ae81-ab6568a965c4\") " Oct 30 21:07:41.500840 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.499601 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94768319-93cc-4cc4-ae81-ab6568a965c4-openshift-service-ca" pod "94768319-93cc-4cc4-ae81-ab6568a965c4" (UID: "94768319-93cc-4cc4-ae81-ab6568a965c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:07:41.506165 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.505324 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94768319-93cc-4cc4-ae81-ab6568a965c4-metrics-certs" pod "94768319-93cc-4cc4-ae81-ab6568a965c4" (UID: "94768319-93cc-4cc4-ae81-ab6568a965c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:07:41.533622 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.533573 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94768319-93cc-4cc4-ae81-ab6568a965c4-maas-api-tls" pod "94768319-93cc-4cc4-ae81-ab6568a965c4" (UID: "94768319-93cc-4cc4-ae81-ab6568a965c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:07:41.534264 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.534232 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94768319-93cc-4cc4-ae81-ab6568a965c4-kube-api-access-cvx6p" pod "94768319-93cc-4cc4-ae81-ab6568a965c4" (UID: "94768319-93cc-4cc4-ae81-ab6568a965c4"). InnerVolumeSpecName "kube-api-access-cvx6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:07:41.599729 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.599494 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94768319-93cc-4cc4-ae81-ab6568a965c4-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:41.599729 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.599516 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94768319-93cc-4cc4-ae81-ab6568a965c4-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:41.599729 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.599529 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94768319-93cc-4cc4-ae81-ab6568a965c4-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:41.599729 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.599539 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvx6p\" (UniqueName: \"kubernetes.io/projected/94768319-93cc-4cc4-ae81-ab6568a965c4-kube-api-access-cvx6p\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:41.689654 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.689625 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd"] Oct 30 21:07:41.691223 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:07:41.691189 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod127e9b91_a76e_46f6_9970_e5dcf8a0ba78.slice/crio-cde5c62430111eac3bc5ad35dfb2673c2d95a91ba28288acf446eaced4860d96 WatchSource:0}: Error finding container cde5c62430111eac3bc5ad35dfb2673c2d95a91ba28288acf446eaced4860d96: Status 404 returned error can't find the container with id cde5c62430111eac3bc5ad35dfb2673c2d95a91ba28288acf446eaced4860d96 Oct 30 21:07:41.694148 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.694114 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:07:41.694234 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.694182 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:07:41.694234 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:41.694212 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:07:42.112239 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:42.112151 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" Oct 30 21:07:42.112239 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:42.112169 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45" event={"ID":"94768319-93cc-4cc4-ae81-ab6568a965c4","Type":"ContainerDied","Data":"c1ce210dcb8e1faf6fac459208212aee6bf7abbe807ee986592ce577418f14b2"} Oct 30 21:07:42.112464 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:42.112247 4848 scope.go:122] "RemoveContainer" containerID="5f5d8a64c68300654efd39a789829c13a8b520e5814bd99cb886680676511662" Oct 30 21:07:42.114186 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:42.114131 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" event={"ID":"127e9b91-a76e-46f6-9970-e5dcf8a0ba78","Type":"ContainerStarted","Data":"623d607cb97ef2790866d8baa631053d6a9e8f0400dd5e4ac0d59e36cc83849f"} Oct 30 21:07:42.114324 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:42.114214 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" event={"ID":"127e9b91-a76e-46f6-9970-e5dcf8a0ba78","Type":"ContainerStarted","Data":"cde5c62430111eac3bc5ad35dfb2673c2d95a91ba28288acf446eaced4860d96"} Oct 30 21:07:42.176790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:42.176297 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" podStartSLOduration=2.176273067 podStartE2EDuration="2.176273067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:42.157283859 +0000 UTC m=+1767.337180394" watchObservedRunningTime="2026-10-30 21:07:42.176273067 +0000 UTC m=+1767.356169576" Oct 30 21:07:42.227462 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:42.227431 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45"] Oct 30 21:07:42.230867 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:42.230838 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-89d993-97b69c496-crq45"] Oct 30 21:07:42.401245 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:42.401143 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:42.407602 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:42.407575 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:42.939203 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:42.939155 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94768319-93cc-4cc4-ae81-ab6568a965c4" path="/var/lib/kubelet/pods/94768319-93cc-4cc4-ae81-ab6568a965c4/volumes" Oct 30 21:07:43.123004 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:43.122973 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:43.123972 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:43.123953 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:07:44.504511 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:44.504479 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72" Oct 30 21:07:44.552177 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:44.552152 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc" Oct 30 21:07:44.664198 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:44.664168 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/711bf26e-5e4f-4b03-93c2-5302e22b543d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711bf26e-5e4f-4b03-93c2-5302e22b543d-plugins-config-volume\") pod \"711bf26e-5e4f-4b03-93c2-5302e22b543d\" (UID: \"711bf26e-5e4f-4b03-93c2-5302e22b543d\") " Oct 30 21:07:44.664384 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:44.664249 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd1bad1a-2f48-4b2e-801d-ad3762be02c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd1bad1a-2f48-4b2e-801d-ad3762be02c8-plugins-config-volume\") pod \"fd1bad1a-2f48-4b2e-801d-ad3762be02c8\" (UID: \"fd1bad1a-2f48-4b2e-801d-ad3762be02c8\") " Oct 30 21:07:44.664431 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:44.664385 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711bf26e-5e4f-4b03-93c2-5302e22b543d-kube-api-access-t4zgs\" (UniqueName: \"kubernetes.io/projected/711bf26e-5e4f-4b03-93c2-5302e22b543d-kube-api-access-t4zgs\") pod \"711bf26e-5e4f-4b03-93c2-5302e22b543d\" (UID: \"711bf26e-5e4f-4b03-93c2-5302e22b543d\") " Oct 30 21:07:44.664571 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:44.664553 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd1bad1a-2f48-4b2e-801d-ad3762be02c8-kube-api-access-t926g\" (UniqueName: \"kubernetes.io/projected/fd1bad1a-2f48-4b2e-801d-ad3762be02c8-kube-api-access-t926g\") pod \"fd1bad1a-2f48-4b2e-801d-ad3762be02c8\" (UID: \"fd1bad1a-2f48-4b2e-801d-ad3762be02c8\") " Oct 30 21:07:44.664644 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:44.664617 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1bad1a-2f48-4b2e-801d-ad3762be02c8-plugins-config-volume" pod "fd1bad1a-2f48-4b2e-801d-ad3762be02c8" (UID: "fd1bad1a-2f48-4b2e-801d-ad3762be02c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:07:44.664684 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:44.664623 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711bf26e-5e4f-4b03-93c2-5302e22b543d-plugins-config-volume" pod "711bf26e-5e4f-4b03-93c2-5302e22b543d" (UID: "711bf26e-5e4f-4b03-93c2-5302e22b543d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:07:44.665145 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:44.665125 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/711bf26e-5e4f-4b03-93c2-5302e22b543d-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:44.665145 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:44.665146 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd1bad1a-2f48-4b2e-801d-ad3762be02c8-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:44.666430 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:44.666407 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711bf26e-5e4f-4b03-93c2-5302e22b543d-kube-api-access-t4zgs" pod "711bf26e-5e4f-4b03-93c2-5302e22b543d" (UID: "711bf26e-5e4f-4b03-93c2-5302e22b543d"). InnerVolumeSpecName "kube-api-access-t4zgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:07:44.666652 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:44.666629 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1bad1a-2f48-4b2e-801d-ad3762be02c8-kube-api-access-t926g" pod "fd1bad1a-2f48-4b2e-801d-ad3762be02c8" (UID: "fd1bad1a-2f48-4b2e-801d-ad3762be02c8"). InnerVolumeSpecName "kube-api-access-t926g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:07:44.766357 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:44.766272 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4zgs\" (UniqueName: \"kubernetes.io/projected/711bf26e-5e4f-4b03-93c2-5302e22b543d-kube-api-access-t4zgs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:44.766357 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:44.766303 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t926g\" (UniqueName: \"kubernetes.io/projected/fd1bad1a-2f48-4b2e-801d-ad3762be02c8-kube-api-access-t926g\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:45.137846 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:45.137732 4848 generic.go:356] "Generic (PLEG): container finished" podID="fd1bad1a-2f48-4b2e-801d-ad3762be02c8" containerID="32cd53445921a9c54c082566c8a2a58114f3e457a82c649381c5766a9b057513" exitCode=0 Oct 30 21:07:45.137846 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:45.137824 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72" Oct 30 21:07:45.137846 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:45.137832 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72" event={"ID":"fd1bad1a-2f48-4b2e-801d-ad3762be02c8","Type":"ContainerDied","Data":"32cd53445921a9c54c082566c8a2a58114f3e457a82c649381c5766a9b057513"} Oct 30 21:07:45.138149 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:45.137888 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72" event={"ID":"fd1bad1a-2f48-4b2e-801d-ad3762be02c8","Type":"ContainerDied","Data":"0cbe0f8529351bf415e028c8af13c5a39acbdbac3e99fecacd957f3e8df9d8de"} Oct 30 21:07:45.138149 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:45.137902 4848 scope.go:122] "RemoveContainer" containerID="32cd53445921a9c54c082566c8a2a58114f3e457a82c649381c5766a9b057513" Oct 30 21:07:45.139475 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:45.139448 4848 generic.go:356] "Generic (PLEG): container finished" podID="711bf26e-5e4f-4b03-93c2-5302e22b543d" containerID="3212d060a18b1c6e5606169a0485e2f8a6660349c3e3cf2e6557effdbaaf73dc" exitCode=0 Oct 30 21:07:45.139612 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:45.139576 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc" event={"ID":"711bf26e-5e4f-4b03-93c2-5302e22b543d","Type":"ContainerDied","Data":"3212d060a18b1c6e5606169a0485e2f8a6660349c3e3cf2e6557effdbaaf73dc"} Oct 30 21:07:45.139676 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:45.139631 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc" event={"ID":"711bf26e-5e4f-4b03-93c2-5302e22b543d","Type":"ContainerDied","Data":"b220cde6bc2aa1d6ca96b5d329d804224654b98b177f2e02fc1e38caf6f554a0"} Oct 30 21:07:45.139676 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:45.139659 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc" Oct 30 21:07:45.148761 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:45.148736 4848 scope.go:122] "RemoveContainer" containerID="32cd53445921a9c54c082566c8a2a58114f3e457a82c649381c5766a9b057513" Oct 30 21:07:45.149073 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:07:45.149055 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32cd53445921a9c54c082566c8a2a58114f3e457a82c649381c5766a9b057513\": container with ID starting with 32cd53445921a9c54c082566c8a2a58114f3e457a82c649381c5766a9b057513 not found: ID does not exist" containerID="32cd53445921a9c54c082566c8a2a58114f3e457a82c649381c5766a9b057513" Oct 30 21:07:45.149143 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:45.149085 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32cd53445921a9c54c082566c8a2a58114f3e457a82c649381c5766a9b057513"} err="failed to get container status \"32cd53445921a9c54c082566c8a2a58114f3e457a82c649381c5766a9b057513\": rpc error: code = NotFound desc = could not find container \"32cd53445921a9c54c082566c8a2a58114f3e457a82c649381c5766a9b057513\": container with ID starting with 32cd53445921a9c54c082566c8a2a58114f3e457a82c649381c5766a9b057513 not found: ID does not exist" Oct 30 21:07:45.149143 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:45.149102 4848 scope.go:122] "RemoveContainer" containerID="3212d060a18b1c6e5606169a0485e2f8a6660349c3e3cf2e6557effdbaaf73dc" Oct 30 21:07:45.159862 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:45.159838 4848 scope.go:122] "RemoveContainer" containerID="3212d060a18b1c6e5606169a0485e2f8a6660349c3e3cf2e6557effdbaaf73dc" Oct 30 21:07:45.160192 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:07:45.160170 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3212d060a18b1c6e5606169a0485e2f8a6660349c3e3cf2e6557effdbaaf73dc\": container with ID starting with 3212d060a18b1c6e5606169a0485e2f8a6660349c3e3cf2e6557effdbaaf73dc not found: ID does not exist" containerID="3212d060a18b1c6e5606169a0485e2f8a6660349c3e3cf2e6557effdbaaf73dc" Oct 30 21:07:45.160258 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:45.160199 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3212d060a18b1c6e5606169a0485e2f8a6660349c3e3cf2e6557effdbaaf73dc"} err="failed to get container status \"3212d060a18b1c6e5606169a0485e2f8a6660349c3e3cf2e6557effdbaaf73dc\": rpc error: code = NotFound desc = could not find container \"3212d060a18b1c6e5606169a0485e2f8a6660349c3e3cf2e6557effdbaaf73dc\": container with ID starting with 3212d060a18b1c6e5606169a0485e2f8a6660349c3e3cf2e6557effdbaaf73dc not found: ID does not exist" Oct 30 21:07:45.160883 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:45.160865 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72"] Oct 30 21:07:45.169414 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:45.169390 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-89d993-c6c97799f-hhx72"] Oct 30 21:07:45.182356 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:45.182274 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc"] Oct 30 21:07:45.186397 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:45.186373 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-89d993-775594f7d7-b6dpc"] Oct 30 21:07:46.955778 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:46.955731 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="711bf26e-5e4f-4b03-93c2-5302e22b543d" path="/var/lib/kubelet/pods/711bf26e-5e4f-4b03-93c2-5302e22b543d/volumes" Oct 30 21:07:46.956275 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:46.956258 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd1bad1a-2f48-4b2e-801d-ad3762be02c8" path="/var/lib/kubelet/pods/fd1bad1a-2f48-4b2e-801d-ad3762be02c8/volumes" Oct 30 21:07:51.100786 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:51.096450 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c"] Oct 30 21:07:51.100786 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:51.096955 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" podUID="c8b6880d-49f5-4900-8bfb-6d7332195be4" containerName="istio-proxy" containerID="cri-o://486ca2cf193f5a0fbacdb3d1260439c213c223c2770ffaf2340ec3d859d10c87" gracePeriod=30 Oct 30 21:07:52.575517 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:07:52.575472 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:07:54.760104 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:54.760064 4848 patch_prober.go:29] interesting pod/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:07:54.760570 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:54.760138 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" podUID="c8b6880d-49f5-4900-8bfb-6d7332195be4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:07:56.702735 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.702703 4848 generic.go:356] "Generic (PLEG): container finished" podID="c8b6880d-49f5-4900-8bfb-6d7332195be4" containerID="486ca2cf193f5a0fbacdb3d1260439c213c223c2770ffaf2340ec3d859d10c87" exitCode=0 Oct 30 21:07:56.703207 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.702786 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" event={"ID":"c8b6880d-49f5-4900-8bfb-6d7332195be4","Type":"ContainerDied","Data":"486ca2cf193f5a0fbacdb3d1260439c213c223c2770ffaf2340ec3d859d10c87"} Oct 30 21:07:56.703207 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.702884 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" event={"ID":"c8b6880d-49f5-4900-8bfb-6d7332195be4","Type":"ContainerDied","Data":"7d1b83499838dde9174a01134b60f9080262eac3ea753c655c9901b8d27b7805"} Oct 30 21:07:56.703207 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.702912 4848 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d1b83499838dde9174a01134b60f9080262eac3ea753c655c9901b8d27b7805" Oct 30 21:07:56.705596 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.705578 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:56.791154 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.791121 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-workload-socket\") pod \"c8b6880d-49f5-4900-8bfb-6d7332195be4\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " Oct 30 21:07:56.791359 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.791177 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-podinfo\") pod \"c8b6880d-49f5-4900-8bfb-6d7332195be4\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " Oct 30 21:07:56.791359 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.791246 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b6880d-49f5-4900-8bfb-6d7332195be4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b6880d-49f5-4900-8bfb-6d7332195be4-istiod-ca-cert\") pod \"c8b6880d-49f5-4900-8bfb-6d7332195be4\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " Oct 30 21:07:56.791478 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.791435 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-workload-certs\") pod \"c8b6880d-49f5-4900-8bfb-6d7332195be4\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " Oct 30 21:07:56.791573 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.791441 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-workload-socket" pod "c8b6880d-49f5-4900-8bfb-6d7332195be4" (UID: "c8b6880d-49f5-4900-8bfb-6d7332195be4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:07:56.791631 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.791562 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-data\") pod \"c8b6880d-49f5-4900-8bfb-6d7332195be4\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " Oct 30 21:07:56.791682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.791631 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-workload-certs" pod "c8b6880d-49f5-4900-8bfb-6d7332195be4" (UID: "c8b6880d-49f5-4900-8bfb-6d7332195be4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:07:56.791682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.791638 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b6880d-49f5-4900-8bfb-6d7332195be4-istiod-ca-cert" pod "c8b6880d-49f5-4900-8bfb-6d7332195be4" (UID: "c8b6880d-49f5-4900-8bfb-6d7332195be4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:07:56.791682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.791648 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-token\" (UniqueName: \"kubernetes.io/projected/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-token\") pod \"c8b6880d-49f5-4900-8bfb-6d7332195be4\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " Oct 30 21:07:56.791853 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.791685 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b6880d-49f5-4900-8bfb-6d7332195be4-kube-api-access-6kjrz\" (UniqueName: \"kubernetes.io/projected/c8b6880d-49f5-4900-8bfb-6d7332195be4-kube-api-access-6kjrz\") pod \"c8b6880d-49f5-4900-8bfb-6d7332195be4\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " Oct 30 21:07:56.791853 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.791793 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-data" pod "c8b6880d-49f5-4900-8bfb-6d7332195be4" (UID: "c8b6880d-49f5-4900-8bfb-6d7332195be4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:07:56.791959 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.791879 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-envoy\") pod \"c8b6880d-49f5-4900-8bfb-6d7332195be4\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " Oct 30 21:07:56.791959 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.791921 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-credential-socket\") pod \"c8b6880d-49f5-4900-8bfb-6d7332195be4\" (UID: \"c8b6880d-49f5-4900-8bfb-6d7332195be4\") " Oct 30 21:07:56.792297 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.792273 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-credential-socket" pod "c8b6880d-49f5-4900-8bfb-6d7332195be4" (UID: "c8b6880d-49f5-4900-8bfb-6d7332195be4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:07:56.792640 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.792618 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:56.792737 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.792646 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b6880d-49f5-4900-8bfb-6d7332195be4-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:56.792737 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.792660 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:56.792737 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.792675 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:56.792737 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.792691 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:56.794023 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.793963 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b6880d-49f5-4900-8bfb-6d7332195be4-kube-api-access-6kjrz" pod "c8b6880d-49f5-4900-8bfb-6d7332195be4" (UID: "c8b6880d-49f5-4900-8bfb-6d7332195be4"). InnerVolumeSpecName "kube-api-access-6kjrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:07:56.794226 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.794198 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-podinfo" pod "c8b6880d-49f5-4900-8bfb-6d7332195be4" (UID: "c8b6880d-49f5-4900-8bfb-6d7332195be4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:07:56.794306 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.794256 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-token" pod "c8b6880d-49f5-4900-8bfb-6d7332195be4" (UID: "c8b6880d-49f5-4900-8bfb-6d7332195be4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:07:56.794719 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.794702 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-envoy" pod "c8b6880d-49f5-4900-8bfb-6d7332195be4" (UID: "c8b6880d-49f5-4900-8bfb-6d7332195be4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:07:56.893952 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.893872 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:56.893952 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.893902 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kjrz\" (UniqueName: \"kubernetes.io/projected/c8b6880d-49f5-4900-8bfb-6d7332195be4-kube-api-access-6kjrz\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:56.893952 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.893918 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:56.893952 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:56.893927 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b6880d-49f5-4900-8bfb-6d7332195be4-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:07:57.709438 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:57.709405 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c" Oct 30 21:07:57.733995 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:57.733963 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c"] Oct 30 21:07:57.737554 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:57.737532 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-6538cc-openshift-default-6f4b9959b5-kq47c"] Oct 30 21:07:58.934151 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:07:58.934119 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b6880d-49f5-4900-8bfb-6d7332195be4" path="/var/lib/kubelet/pods/c8b6880d-49f5-4900-8bfb-6d7332195be4/volumes" Oct 30 21:08:00.548617 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.548579 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq"] Oct 30 21:08:00.549684 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.549660 4848 cpu_manager.go:414] "containerMap: removing container" podUID="c8b6880d-49f5-4900-8bfb-6d7332195be4" containerName="istio-proxy" Oct 30 21:08:00.549830 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.549687 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b6880d-49f5-4900-8bfb-6d7332195be4" containerName="istio-proxy" Oct 30 21:08:00.549830 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.549718 4848 cpu_manager.go:414] "containerMap: removing container" podUID="711bf26e-5e4f-4b03-93c2-5302e22b543d" containerName="payload-pre-processing" Oct 30 21:08:00.549830 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.549727 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711bf26e-5e4f-4b03-93c2-5302e22b543d" containerName="payload-pre-processing" Oct 30 21:08:00.549830 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.549750 4848 cpu_manager.go:414] "containerMap: removing container" podUID="94768319-93cc-4cc4-ae81-ab6568a965c4" containerName="maas-api" Oct 30 21:08:00.549830 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.549758 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94768319-93cc-4cc4-ae81-ab6568a965c4" containerName="maas-api" Oct 30 21:08:00.550117 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.549874 4848 cpu_manager.go:414] "containerMap: removing container" podUID="fd1bad1a-2f48-4b2e-801d-ad3762be02c8" containerName="payload-processing" Oct 30 21:08:00.550117 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.549885 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd1bad1a-2f48-4b2e-801d-ad3762be02c8" containerName="payload-processing" Oct 30 21:08:00.550117 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.550071 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd1bad1a-2f48-4b2e-801d-ad3762be02c8" containerName="payload-processing" Oct 30 21:08:00.550117 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.550092 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b6880d-49f5-4900-8bfb-6d7332195be4" containerName="istio-proxy" Oct 30 21:08:00.550117 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.550108 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="711bf26e-5e4f-4b03-93c2-5302e22b543d" containerName="payload-pre-processing" Oct 30 21:08:00.550117 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.550119 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="94768319-93cc-4cc4-ae81-ab6568a965c4" containerName="maas-api" Oct 30 21:08:00.555799 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.555760 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq"] Oct 30 21:08:00.556042 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.556016 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" Oct 30 21:08:00.559934 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.559912 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-99668e-serving-cert\"" Oct 30 21:08:00.656897 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.647280 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2d7d\" (UniqueName: \"kubernetes.io/projected/813afec3-2ef0-434f-b4bd-08c9c705c6ec-kube-api-access-l2d7d\") pod \"maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq\" (UID: \"813afec3-2ef0-434f-b4bd-08c9c705c6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" Oct 30 21:08:00.656897 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.647357 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/813afec3-2ef0-434f-b4bd-08c9c705c6ec-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq\" (UID: \"813afec3-2ef0-434f-b4bd-08c9c705c6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" Oct 30 21:08:00.656897 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.647396 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/813afec3-2ef0-434f-b4bd-08c9c705c6ec-maas-api-tls\") pod \"maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq\" (UID: \"813afec3-2ef0-434f-b4bd-08c9c705c6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" Oct 30 21:08:00.656897 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.647455 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/813afec3-2ef0-434f-b4bd-08c9c705c6ec-metrics-certs\") pod \"maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq\" (UID: \"813afec3-2ef0-434f-b4bd-08c9c705c6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" Oct 30 21:08:00.764789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.763949 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2d7d\" (UniqueName: \"kubernetes.io/projected/813afec3-2ef0-434f-b4bd-08c9c705c6ec-kube-api-access-l2d7d\") pod \"maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq\" (UID: \"813afec3-2ef0-434f-b4bd-08c9c705c6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" Oct 30 21:08:00.764789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.764043 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/813afec3-2ef0-434f-b4bd-08c9c705c6ec-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq\" (UID: \"813afec3-2ef0-434f-b4bd-08c9c705c6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" Oct 30 21:08:00.764789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.764083 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/813afec3-2ef0-434f-b4bd-08c9c705c6ec-maas-api-tls\") pod \"maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq\" (UID: \"813afec3-2ef0-434f-b4bd-08c9c705c6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" Oct 30 21:08:00.764789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.764144 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/813afec3-2ef0-434f-b4bd-08c9c705c6ec-metrics-certs\") pod \"maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq\" (UID: \"813afec3-2ef0-434f-b4bd-08c9c705c6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" Oct 30 21:08:00.765509 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.765485 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/813afec3-2ef0-434f-b4bd-08c9c705c6ec-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq\" (UID: \"813afec3-2ef0-434f-b4bd-08c9c705c6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" Oct 30 21:08:00.767651 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.767628 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/813afec3-2ef0-434f-b4bd-08c9c705c6ec-metrics-certs\") pod \"maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq\" (UID: \"813afec3-2ef0-434f-b4bd-08c9c705c6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" Oct 30 21:08:00.767804 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.767761 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/813afec3-2ef0-434f-b4bd-08c9c705c6ec-maas-api-tls\") pod \"maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq\" (UID: \"813afec3-2ef0-434f-b4bd-08c9c705c6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" Oct 30 21:08:00.772822 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.772759 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2d7d\" (UniqueName: \"kubernetes.io/projected/813afec3-2ef0-434f-b4bd-08c9c705c6ec-kube-api-access-l2d7d\") pod \"maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq\" (UID: \"813afec3-2ef0-434f-b4bd-08c9c705c6ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" Oct 30 21:08:00.870314 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:00.870237 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" Oct 30 21:08:01.265987 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:01.261262 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq"] Oct 30 21:08:01.797494 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:01.797464 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" event={"ID":"813afec3-2ef0-434f-b4bd-08c9c705c6ec","Type":"ContainerStarted","Data":"02647024efeae4bf4bfc0ad39f3c483588d173b4874ab3abd3ae84e662d2ddf2"} Oct 30 21:08:02.833873 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:02.833008 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" event={"ID":"813afec3-2ef0-434f-b4bd-08c9c705c6ec","Type":"ContainerStarted","Data":"ce9698fee3857a1172705c68432d57ff35079040aa20506e488792140aa175a1"} Oct 30 21:08:02.833873 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:02.833079 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" Oct 30 21:08:02.854096 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:02.854045 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" podStartSLOduration=2.363868738 podStartE2EDuration="2.85403179s" totalImagesPullingTime="490.163052ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:02.850682284 +0000 UTC m=+1788.030578791" watchObservedRunningTime="2026-10-30 21:08:02.85403179 +0000 UTC m=+1788.033928632" Oct 30 21:08:03.732859 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:03.732819 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk"] Oct 30 21:08:03.733352 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:03.733317 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" podUID="672d778a-896f-479e-877f-473372d7456c" containerName="istio-proxy" containerID="cri-o://3710606c84f011220dc73ec01e856f2084e59d2cc086aac1518f2854ee7c3a1c" gracePeriod=30 Oct 30 21:08:03.738616 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:03.738583 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp"] Oct 30 21:08:03.738831 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:03.738809 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" podUID="f7cbc43b-dad3-4ab6-ad12-e03522a725b0" containerName="istio-proxy" containerID="cri-o://a9b066e264757d53ee38596b8c4892f594b83d461de429ae76a63c632c11b340" gracePeriod=30 Oct 30 21:08:03.742427 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:03.742396 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt"] Oct 30 21:08:03.742675 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:03.742615 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" podUID="087bab2d-59bc-4d7e-a526-2d84d6a2954f" containerName="istio-proxy" containerID="cri-o://ff6c870679bf367d52efb602a2ae70bbc8a9b8e5f3d529d953b6019b9541e344" gracePeriod=30 Oct 30 21:08:05.966012 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:05.965970 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv"] Oct 30 21:08:08.855600 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:08.855578 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" Oct 30 21:08:08.926221 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:08.926188 4848 generic.go:356] "Generic (PLEG): container finished" podID="f7cbc43b-dad3-4ab6-ad12-e03522a725b0" containerID="a9b066e264757d53ee38596b8c4892f594b83d461de429ae76a63c632c11b340" exitCode=0 Oct 30 21:08:08.926412 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:08.926280 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" event={"ID":"f7cbc43b-dad3-4ab6-ad12-e03522a725b0","Type":"ContainerDied","Data":"a9b066e264757d53ee38596b8c4892f594b83d461de429ae76a63c632c11b340"} Oct 30 21:08:08.927983 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:08.927958 4848 generic.go:356] "Generic (PLEG): container finished" podID="672d778a-896f-479e-877f-473372d7456c" containerID="3710606c84f011220dc73ec01e856f2084e59d2cc086aac1518f2854ee7c3a1c" exitCode=0 Oct 30 21:08:08.928177 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:08.928160 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" event={"ID":"672d778a-896f-479e-877f-473372d7456c","Type":"ContainerDied","Data":"3710606c84f011220dc73ec01e856f2084e59d2cc086aac1518f2854ee7c3a1c"} Oct 30 21:08:08.931950 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:08.931912 4848 generic.go:356] "Generic (PLEG): container finished" podID="087bab2d-59bc-4d7e-a526-2d84d6a2954f" containerID="ff6c870679bf367d52efb602a2ae70bbc8a9b8e5f3d529d953b6019b9541e344" exitCode=0 Oct 30 21:08:08.954850 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:08.954813 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" event={"ID":"087bab2d-59bc-4d7e-a526-2d84d6a2954f","Type":"ContainerDied","Data":"ff6c870679bf367d52efb602a2ae70bbc8a9b8e5f3d529d953b6019b9541e344"} Oct 30 21:08:09.164752 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.164620 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:08:09.179094 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.179069 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/672d778a-896f-479e-877f-473372d7456c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/672d778a-896f-479e-877f-473372d7456c-istiod-ca-cert\") pod \"672d778a-896f-479e-877f-473372d7456c\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " Oct 30 21:08:09.179631 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.179611 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-workload-socket\") pod \"672d778a-896f-479e-877f-473372d7456c\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " Oct 30 21:08:09.179735 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.179678 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-istio-envoy\") pod \"672d778a-896f-479e-877f-473372d7456c\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " Oct 30 21:08:09.179735 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.179700 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/672d778a-896f-479e-877f-473372d7456c-istio-token\" (UniqueName: \"kubernetes.io/projected/672d778a-896f-479e-877f-473372d7456c-istio-token\") pod \"672d778a-896f-479e-877f-473372d7456c\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " Oct 30 21:08:09.179880 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.179736 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/672d778a-896f-479e-877f-473372d7456c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/672d778a-896f-479e-877f-473372d7456c-istio-podinfo\") pod \"672d778a-896f-479e-877f-473372d7456c\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " Oct 30 21:08:09.179880 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.179804 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/672d778a-896f-479e-877f-473372d7456c-kube-api-access-bcm9s\" (UniqueName: \"kubernetes.io/projected/672d778a-896f-479e-877f-473372d7456c-kube-api-access-bcm9s\") pod \"672d778a-896f-479e-877f-473372d7456c\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " Oct 30 21:08:09.179996 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.179901 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-workload-certs\") pod \"672d778a-896f-479e-877f-473372d7456c\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " Oct 30 21:08:09.180836 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.179500 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/672d778a-896f-479e-877f-473372d7456c-istiod-ca-cert" pod "672d778a-896f-479e-877f-473372d7456c" (UID: "672d778a-896f-479e-877f-473372d7456c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:08:09.180836 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.180606 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-workload-certs" pod "672d778a-896f-479e-877f-473372d7456c" (UID: "672d778a-896f-479e-877f-473372d7456c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:09.180836 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.180801 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-workload-socket" pod "672d778a-896f-479e-877f-473372d7456c" (UID: "672d778a-896f-479e-877f-473372d7456c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:09.180836 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.180808 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-istio-data" pod "672d778a-896f-479e-877f-473372d7456c" (UID: "672d778a-896f-479e-877f-473372d7456c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:09.181296 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.181271 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-istio-data\") pod \"672d778a-896f-479e-877f-473372d7456c\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " Oct 30 21:08:09.181569 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.181550 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-credential-socket\") pod \"672d778a-896f-479e-877f-473372d7456c\" (UID: \"672d778a-896f-479e-877f-473372d7456c\") " Oct 30 21:08:09.182128 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.181865 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-credential-socket" pod "672d778a-896f-479e-877f-473372d7456c" (UID: "672d778a-896f-479e-877f-473372d7456c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:09.183683 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.183667 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.183874 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.183836 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.183874 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.183857 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.183874 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.183872 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/672d778a-896f-479e-877f-473372d7456c-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.184164 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.183886 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.184390 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.184363 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/672d778a-896f-479e-877f-473372d7456c-istio-podinfo" pod "672d778a-896f-479e-877f-473372d7456c" (UID: "672d778a-896f-479e-877f-473372d7456c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:08:09.191972 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.188478 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672d778a-896f-479e-877f-473372d7456c-istio-token" pod "672d778a-896f-479e-877f-473372d7456c" (UID: "672d778a-896f-479e-877f-473372d7456c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:08:09.205646 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.205605 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672d778a-896f-479e-877f-473372d7456c-kube-api-access-bcm9s" pod "672d778a-896f-479e-877f-473372d7456c" (UID: "672d778a-896f-479e-877f-473372d7456c"). InnerVolumeSpecName "kube-api-access-bcm9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:08:09.205876 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.205833 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-istio-envoy" pod "672d778a-896f-479e-877f-473372d7456c" (UID: "672d778a-896f-479e-877f-473372d7456c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:09.230205 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.230170 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:08:09.231657 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.231630 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:08:09.284803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.284718 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-kube-api-access-lkgrw\" (UniqueName: \"kubernetes.io/projected/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-kube-api-access-lkgrw\") pod \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " Oct 30 21:08:09.285050 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.285029 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-credential-socket\") pod \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " Oct 30 21:08:09.285205 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.285188 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-data\") pod \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " Oct 30 21:08:09.285335 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.285320 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/087bab2d-59bc-4d7e-a526-2d84d6a2954f-kube-api-access-c5zd7\" (UniqueName: \"kubernetes.io/projected/087bab2d-59bc-4d7e-a526-2d84d6a2954f-kube-api-access-c5zd7\") pod \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " Oct 30 21:08:09.285490 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.285473 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-workload-socket\") pod \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " Oct 30 21:08:09.285686 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.285668 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-token\" (UniqueName: \"kubernetes.io/projected/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-token\") pod \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " Oct 30 21:08:09.285865 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.285842 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-envoy\") pod \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " Oct 30 21:08:09.285955 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.285890 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istiod-ca-cert\") pod \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " Oct 30 21:08:09.285955 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.285920 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-envoy\") pod \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " Oct 30 21:08:09.286106 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.286092 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-workload-socket\") pod \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " Oct 30 21:08:09.286173 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.286131 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-credential-socket\") pod \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " Oct 30 21:08:09.286173 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.286165 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-data\") pod \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " Oct 30 21:08:09.286285 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.286226 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-token\" (UniqueName: \"kubernetes.io/projected/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-token\") pod \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " Oct 30 21:08:09.286285 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.286272 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-podinfo\") pod \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " Oct 30 21:08:09.286391 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.286323 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-workload-certs\") pod \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " Oct 30 21:08:09.286391 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.286358 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istiod-ca-cert\") pod \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\" (UID: \"087bab2d-59bc-4d7e-a526-2d84d6a2954f\") " Oct 30 21:08:09.286492 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.286405 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-podinfo\") pod \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " Oct 30 21:08:09.286492 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.286475 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-workload-certs\") pod \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\" (UID: \"f7cbc43b-dad3-4ab6-ad12-e03522a725b0\") " Oct 30 21:08:09.287121 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.287101 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/672d778a-896f-479e-877f-473372d7456c-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.287218 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.287123 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/672d778a-896f-479e-877f-473372d7456c-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.287218 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.287137 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/672d778a-896f-479e-877f-473372d7456c-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.287218 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.287151 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcm9s\" (UniqueName: \"kubernetes.io/projected/672d778a-896f-479e-877f-473372d7456c-kube-api-access-bcm9s\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.287933 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.285377 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-credential-socket" pod "f7cbc43b-dad3-4ab6-ad12-e03522a725b0" (UID: "f7cbc43b-dad3-4ab6-ad12-e03522a725b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:09.288055 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.285564 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-data" pod "087bab2d-59bc-4d7e-a526-2d84d6a2954f" (UID: "087bab2d-59bc-4d7e-a526-2d84d6a2954f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:09.288055 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.287854 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-workload-certs" pod "f7cbc43b-dad3-4ab6-ad12-e03522a725b0" (UID: "f7cbc43b-dad3-4ab6-ad12-e03522a725b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:09.288179 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.288135 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-workload-socket" pod "f7cbc43b-dad3-4ab6-ad12-e03522a725b0" (UID: "f7cbc43b-dad3-4ab6-ad12-e03522a725b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:09.288606 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.288584 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-data" pod "f7cbc43b-dad3-4ab6-ad12-e03522a725b0" (UID: "f7cbc43b-dad3-4ab6-ad12-e03522a725b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:09.288930 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.288817 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istiod-ca-cert" pod "f7cbc43b-dad3-4ab6-ad12-e03522a725b0" (UID: "f7cbc43b-dad3-4ab6-ad12-e03522a725b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:08:09.289023 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.288878 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-workload-socket" pod "087bab2d-59bc-4d7e-a526-2d84d6a2954f" (UID: "087bab2d-59bc-4d7e-a526-2d84d6a2954f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:09.289090 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.289023 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-credential-socket" pod "087bab2d-59bc-4d7e-a526-2d84d6a2954f" (UID: "087bab2d-59bc-4d7e-a526-2d84d6a2954f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:09.289151 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.289126 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-workload-certs" pod "087bab2d-59bc-4d7e-a526-2d84d6a2954f" (UID: "087bab2d-59bc-4d7e-a526-2d84d6a2954f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:09.289322 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.289305 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istiod-ca-cert" pod "087bab2d-59bc-4d7e-a526-2d84d6a2954f" (UID: "087bab2d-59bc-4d7e-a526-2d84d6a2954f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:08:09.291794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.291739 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-podinfo" pod "087bab2d-59bc-4d7e-a526-2d84d6a2954f" (UID: "087bab2d-59bc-4d7e-a526-2d84d6a2954f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:08:09.292067 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.292042 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-token" pod "f7cbc43b-dad3-4ab6-ad12-e03522a725b0" (UID: "f7cbc43b-dad3-4ab6-ad12-e03522a725b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:08:09.294343 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.294312 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-podinfo" pod "f7cbc43b-dad3-4ab6-ad12-e03522a725b0" (UID: "f7cbc43b-dad3-4ab6-ad12-e03522a725b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:08:09.294710 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.294676 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-envoy" pod "087bab2d-59bc-4d7e-a526-2d84d6a2954f" (UID: "087bab2d-59bc-4d7e-a526-2d84d6a2954f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:09.301214 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.301186 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-kube-api-access-lkgrw" pod "f7cbc43b-dad3-4ab6-ad12-e03522a725b0" (UID: "f7cbc43b-dad3-4ab6-ad12-e03522a725b0"). InnerVolumeSpecName "kube-api-access-lkgrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:08:09.301363 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.301237 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-token" pod "087bab2d-59bc-4d7e-a526-2d84d6a2954f" (UID: "087bab2d-59bc-4d7e-a526-2d84d6a2954f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:08:09.301363 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.301259 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/087bab2d-59bc-4d7e-a526-2d84d6a2954f-kube-api-access-c5zd7" pod "087bab2d-59bc-4d7e-a526-2d84d6a2954f" (UID: "087bab2d-59bc-4d7e-a526-2d84d6a2954f"). InnerVolumeSpecName "kube-api-access-c5zd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:08:09.302756 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.302733 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-envoy" pod "f7cbc43b-dad3-4ab6-ad12-e03522a725b0" (UID: "f7cbc43b-dad3-4ab6-ad12-e03522a725b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:09.413017 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.412976 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.413017 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.413018 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.414241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.413034 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.414241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.413049 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.414241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.413064 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.414241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.413078 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.414241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.413091 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.414241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.413105 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.414241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.413118 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.414241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.413131 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.414241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.413145 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.414241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.413158 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.414241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.413171 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.414241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.413186 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkgrw\" (UniqueName: \"kubernetes.io/projected/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-kube-api-access-lkgrw\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.414241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.413201 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.414241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.413216 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/087bab2d-59bc-4d7e-a526-2d84d6a2954f-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.414241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.413230 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5zd7\" (UniqueName: \"kubernetes.io/projected/087bab2d-59bc-4d7e-a526-2d84d6a2954f-kube-api-access-c5zd7\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.414241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.413245 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7cbc43b-dad3-4ab6-ad12-e03522a725b0-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:09.940749 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.940714 4848 generic.go:356] "Generic (PLEG): container finished" podID="05d85436-702d-4fb3-8e98-220cb26ef773" containerID="353971ad64b4eee065b9e6a70897017bc4cd15f08639b81452e92b2a7de0d4a1" exitCode=0 Oct 30 21:08:09.941260 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.940797 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" event={"ID":"05d85436-702d-4fb3-8e98-220cb26ef773","Type":"ContainerDied","Data":"353971ad64b4eee065b9e6a70897017bc4cd15f08639b81452e92b2a7de0d4a1"} Oct 30 21:08:09.941368 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.941342 4848 scope.go:122] "RemoveContainer" containerID="353971ad64b4eee065b9e6a70897017bc4cd15f08639b81452e92b2a7de0d4a1" Oct 30 21:08:09.942879 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.942860 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" Oct 30 21:08:09.942989 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.942872 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt" event={"ID":"087bab2d-59bc-4d7e-a526-2d84d6a2954f","Type":"ContainerDied","Data":"d18dcec1337513db296d7c75facbe89c72a3830254aef32bd998c091e985f708"} Oct 30 21:08:09.942989 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.942929 4848 scope.go:122] "RemoveContainer" containerID="ff6c870679bf367d52efb602a2ae70bbc8a9b8e5f3d529d953b6019b9541e344" Oct 30 21:08:09.943656 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.943606 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:08:09.943785 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.943686 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:08:09.943785 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.943725 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:08:09.944862 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.944843 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" Oct 30 21:08:09.944967 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.944865 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp" event={"ID":"f7cbc43b-dad3-4ab6-ad12-e03522a725b0","Type":"ContainerDied","Data":"5225e7cac38703db8b658fac39351a2a12029e3fecb426f7e4e8aba7486207b3"} Oct 30 21:08:09.946944 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.946883 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" event={"ID":"672d778a-896f-479e-877f-473372d7456c","Type":"ContainerDied","Data":"6ce1f54b140df4e091263341e502196fa3d0c8f0f1c52262dae42250c16e3ee5"} Oct 30 21:08:09.947062 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.946948 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk" Oct 30 21:08:09.961753 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.961735 4848 scope.go:122] "RemoveContainer" containerID="a9b066e264757d53ee38596b8c4892f594b83d461de429ae76a63c632c11b340" Oct 30 21:08:09.973041 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.973025 4848 scope.go:122] "RemoveContainer" containerID="3710606c84f011220dc73ec01e856f2084e59d2cc086aac1518f2854ee7c3a1c" Oct 30 21:08:09.984052 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.983592 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt"] Oct 30 21:08:09.988645 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:09.988620 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-78skt"] Oct 30 21:08:10.002841 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:10.002813 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk"] Oct 30 21:08:10.015715 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:10.015679 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-nwthk"] Oct 30 21:08:10.036595 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:10.036561 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp"] Oct 30 21:08:10.044546 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:10.044522 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-tw7fp"] Oct 30 21:08:10.939911 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:10.939876 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="087bab2d-59bc-4d7e-a526-2d84d6a2954f" path="/var/lib/kubelet/pods/087bab2d-59bc-4d7e-a526-2d84d6a2954f/volumes" Oct 30 21:08:10.940648 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:10.940617 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="672d778a-896f-479e-877f-473372d7456c" path="/var/lib/kubelet/pods/672d778a-896f-479e-877f-473372d7456c/volumes" Oct 30 21:08:10.941278 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:10.941254 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7cbc43b-dad3-4ab6-ad12-e03522a725b0" path="/var/lib/kubelet/pods/f7cbc43b-dad3-4ab6-ad12-e03522a725b0/volumes" Oct 30 21:08:10.956622 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:10.956593 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" event={"ID":"05d85436-702d-4fb3-8e98-220cb26ef773","Type":"ContainerStarted","Data":"3bbee5d997338eea519f63b8e9493cecc1b6a085cecbc61b37179ff81acb5103"} Oct 30 21:08:11.201715 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:11.201682 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:08:11.203114 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:11.203087 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 21:08:11.203224 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:11.203137 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" podUID="05d85436-702d-4fb3-8e98-220cb26ef773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 21:08:12.202386 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:12.202344 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 21:08:12.202834 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:12.202405 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" podUID="05d85436-702d-4fb3-8e98-220cb26ef773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 21:08:13.202206 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.202162 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 21:08:13.202397 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.202227 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" podUID="05d85436-702d-4fb3-8e98-220cb26ef773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 21:08:13.292886 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.292847 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj"] Oct 30 21:08:13.293931 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.293898 4848 cpu_manager.go:414] "containerMap: removing container" podUID="672d778a-896f-479e-877f-473372d7456c" containerName="istio-proxy" Oct 30 21:08:13.293931 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.293924 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="672d778a-896f-479e-877f-473372d7456c" containerName="istio-proxy" Oct 30 21:08:13.294094 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.293966 4848 cpu_manager.go:414] "containerMap: removing container" podUID="f7cbc43b-dad3-4ab6-ad12-e03522a725b0" containerName="istio-proxy" Oct 30 21:08:13.294094 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.293974 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7cbc43b-dad3-4ab6-ad12-e03522a725b0" containerName="istio-proxy" Oct 30 21:08:13.294094 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.294010 4848 cpu_manager.go:414] "containerMap: removing container" podUID="087bab2d-59bc-4d7e-a526-2d84d6a2954f" containerName="istio-proxy" Oct 30 21:08:13.294094 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.294018 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="087bab2d-59bc-4d7e-a526-2d84d6a2954f" containerName="istio-proxy" Oct 30 21:08:13.294251 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.294186 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="672d778a-896f-479e-877f-473372d7456c" containerName="istio-proxy" Oct 30 21:08:13.294251 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.294202 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7cbc43b-dad3-4ab6-ad12-e03522a725b0" containerName="istio-proxy" Oct 30 21:08:13.294251 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.294219 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="087bab2d-59bc-4d7e-a526-2d84d6a2954f" containerName="istio-proxy" Oct 30 21:08:13.302738 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.302710 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.340821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.340299 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-435414-openshift-default-dockercfg-62smr\"" Oct 30 21:08:13.344427 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.344395 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj"] Oct 30 21:08:13.453842 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.444868 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3402ddb4-1b5f-4458-a868-19a4add51793-istio-podinfo\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.454223 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.454189 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3402ddb4-1b5f-4458-a868-19a4add51793-istio-token\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.457949 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.457911 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-workload-socket\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.458200 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.458179 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsv86\" (UniqueName: \"kubernetes.io/projected/3402ddb4-1b5f-4458-a868-19a4add51793-kube-api-access-rsv86\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.458324 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.458307 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-istio-data\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.471932 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.471893 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3402ddb4-1b5f-4458-a868-19a4add51793-istiod-ca-cert\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.472184 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.472166 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-credential-socket\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.472392 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.472375 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-istio-envoy\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.472524 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.472507 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-workload-certs\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.579824 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.579785 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-istio-envoy\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.580582 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.580557 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-workload-certs\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.581311 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.581290 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3402ddb4-1b5f-4458-a868-19a4add51793-istio-podinfo\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.582017 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.581999 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3402ddb4-1b5f-4458-a868-19a4add51793-istio-token\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.582539 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.582516 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-workload-socket\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.582685 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.582669 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsv86\" (UniqueName: \"kubernetes.io/projected/3402ddb4-1b5f-4458-a868-19a4add51793-kube-api-access-rsv86\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.582833 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.582814 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-istio-data\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.583045 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.583028 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3402ddb4-1b5f-4458-a868-19a4add51793-istiod-ca-cert\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.594110 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.586188 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-workload-socket\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.594110 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.581083 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-workload-certs\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.594110 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.586781 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-istio-data\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.594110 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.587389 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3402ddb4-1b5f-4458-a868-19a4add51793-istiod-ca-cert\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.594861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.594811 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-credential-socket\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.595019 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.595000 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-credential-socket\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.607952 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.607919 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-istio-envoy\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.626887 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.619345 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3402ddb4-1b5f-4458-a868-19a4add51793-istio-token\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.632796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.630310 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsv86\" (UniqueName: \"kubernetes.io/projected/3402ddb4-1b5f-4458-a868-19a4add51793-kube-api-access-rsv86\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.632796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.630812 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3402ddb4-1b5f-4458-a868-19a4add51793-istio-podinfo\") pod \"e2e-autores-435414-openshift-default-7b78b47f57-9lbtj\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.654110 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.654071 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:13.953230 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:13.953191 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:08:14.097430 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:14.097400 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj"] Oct 30 21:08:14.097557 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:08:14.097526 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3402ddb4_1b5f_4458_a868_19a4add51793.slice/crio-0b5cc9a88d503a643135dc7d590a7135654ce4d99a88fbf7f040adc295106a73 WatchSource:0}: Error finding container 0b5cc9a88d503a643135dc7d590a7135654ce4d99a88fbf7f040adc295106a73: Status 404 returned error can't find the container with id 0b5cc9a88d503a643135dc7d590a7135654ce4d99a88fbf7f040adc295106a73 Oct 30 21:08:14.100251 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:14.100223 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:08:14.100340 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:14.100282 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:08:14.100340 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:14.100310 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:08:14.206797 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:14.205153 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 21:08:14.206797 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:14.205213 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" podUID="05d85436-702d-4fb3-8e98-220cb26ef773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 21:08:14.992112 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:14.992077 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" event={"ID":"3402ddb4-1b5f-4458-a868-19a4add51793","Type":"ContainerStarted","Data":"63c93aaaee2974cfa22d1f350b159ccffa635f838191e2bcb40faec6dc5e7126"} Oct 30 21:08:14.992587 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:14.992142 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" event={"ID":"3402ddb4-1b5f-4458-a868-19a4add51793","Type":"ContainerStarted","Data":"0b5cc9a88d503a643135dc7d590a7135654ce4d99a88fbf7f040adc295106a73"} Oct 30 21:08:15.028791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:15.028503 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" podStartSLOduration=2.028482961 podStartE2EDuration="2.028482961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:15.021389128 +0000 UTC m=+1800.201285640" watchObservedRunningTime="2026-10-30 21:08:15.028482961 +0000 UTC m=+1800.208379468" Oct 30 21:08:15.229794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:15.226910 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 21:08:15.229794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:15.226988 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" podUID="05d85436-702d-4fb3-8e98-220cb26ef773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 21:08:15.654668 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:15.654638 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:15.661023 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:15.660996 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:16.013790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:16.013158 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:16.021790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:16.019346 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:08:16.201920 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:16.201887 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 21:08:16.202223 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:16.202194 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" podUID="05d85436-702d-4fb3-8e98-220cb26ef773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 21:08:16.295553 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:16.295469 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 21:08:16.296939 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:16.296920 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 21:08:17.206757 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:17.206718 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:08:17.207753 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:17.207732 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:08:18.627297 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:18.627260 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh"] Oct 30 21:08:18.641941 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:18.641039 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh"] Oct 30 21:08:18.641941 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:18.641165 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh" Oct 30 21:08:18.679595 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:18.678942 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-435414-dockercfg-4777h\"" Oct 30 21:08:18.783800 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:18.783002 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-435414\"" Oct 30 21:08:18.799802 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:18.797818 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh\" (UID: \"a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh" Oct 30 21:08:18.799802 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:18.797978 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtfq5\" (UniqueName: \"kubernetes.io/projected/a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26-kube-api-access-qtfq5\") pod \"payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh\" (UID: \"a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh" Oct 30 21:08:18.907827 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:18.907713 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh\" (UID: \"a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh" Oct 30 21:08:18.909177 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:18.909146 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtfq5\" (UniqueName: \"kubernetes.io/projected/a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26-kube-api-access-qtfq5\") pod \"payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh\" (UID: \"a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh" Oct 30 21:08:18.921547 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:18.908920 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh\" (UID: \"a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh" Oct 30 21:08:18.984749 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:18.984714 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtfq5\" (UniqueName: \"kubernetes.io/projected/a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26-kube-api-access-qtfq5\") pod \"payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh\" (UID: \"a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh" Oct 30 21:08:19.263323 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:19.263279 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh" Oct 30 21:08:19.675178 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:19.672001 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh"] Oct 30 21:08:19.684241 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:08:19.684208 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ccf619_5a46_4593_b3fc_8f3b4fd4ce26.slice/crio-01005e6c83fcfa9d7947c7c165f8c6f68947ae362dd396906a385911f6788bc4 WatchSource:0}: Error finding container 01005e6c83fcfa9d7947c7c165f8c6f68947ae362dd396906a385911f6788bc4: Status 404 returned error can't find the container with id 01005e6c83fcfa9d7947c7c165f8c6f68947ae362dd396906a385911f6788bc4 Oct 30 21:08:20.103285 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:20.103236 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh" event={"ID":"a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26","Type":"ContainerStarted","Data":"19fc0c17524d871cf624e0206ee9a9008076c4440603510885a4f7b433373195"} Oct 30 21:08:20.103489 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:20.103303 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh" event={"ID":"a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26","Type":"ContainerStarted","Data":"01005e6c83fcfa9d7947c7c165f8c6f68947ae362dd396906a385911f6788bc4"} Oct 30 21:08:20.103489 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:20.103377 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh" Oct 30 21:08:20.120209 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:20.120157 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh" podStartSLOduration=2.120143128 podStartE2EDuration="2.120143128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:20.118747751 +0000 UTC m=+1805.298644256" watchObservedRunningTime="2026-10-30 21:08:20.120143128 +0000 UTC m=+1805.300039679" Oct 30 21:08:21.002170 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.002135 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" podUID="9c2f7980-610f-44e9-a586-b48bcf05d315" containerName="main" containerID="cri-o://d858179ca4afb292f2c7084d75a06f3105ba9f7ea6cc98f700a3333beb154318" gracePeriod=45 Oct 30 21:08:21.343525 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.343493 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:08:21.364416 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.364382 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-model-cache\") pod \"9c2f7980-610f-44e9-a586-b48bcf05d315\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " Oct 30 21:08:21.364590 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.364490 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-home\" (UniqueName: \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-home\") pod \"9c2f7980-610f-44e9-a586-b48bcf05d315\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " Oct 30 21:08:21.364590 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.364568 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-dshm\") pod \"9c2f7980-610f-44e9-a586-b48bcf05d315\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " Oct 30 21:08:21.364717 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.364645 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c2f7980-610f-44e9-a586-b48bcf05d315-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2f7980-610f-44e9-a586-b48bcf05d315-tls-certs\") pod \"9c2f7980-610f-44e9-a586-b48bcf05d315\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " Oct 30 21:08:21.364717 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.364651 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-model-cache" pod "9c2f7980-610f-44e9-a586-b48bcf05d315" (UID: "9c2f7980-610f-44e9-a586-b48bcf05d315"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:21.365052 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.365027 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c2f7980-610f-44e9-a586-b48bcf05d315-kube-api-access-2qp8h\" (UniqueName: \"kubernetes.io/projected/9c2f7980-610f-44e9-a586-b48bcf05d315-kube-api-access-2qp8h\") pod \"9c2f7980-610f-44e9-a586-b48bcf05d315\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " Oct 30 21:08:21.365142 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.365100 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-tmp-dir\") pod \"9c2f7980-610f-44e9-a586-b48bcf05d315\" (UID: \"9c2f7980-610f-44e9-a586-b48bcf05d315\") " Oct 30 21:08:21.365200 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.365133 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-home" pod "9c2f7980-610f-44e9-a586-b48bcf05d315" (UID: "9c2f7980-610f-44e9-a586-b48bcf05d315"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:21.365515 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.365496 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-tmp-dir" pod "9c2f7980-610f-44e9-a586-b48bcf05d315" (UID: "9c2f7980-610f-44e9-a586-b48bcf05d315"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:21.365902 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.365876 4848 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-model-cache\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:21.365902 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.365898 4848 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-home\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:21.366076 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.365913 4848 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-tmp-dir\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:21.367257 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.367220 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c2f7980-610f-44e9-a586-b48bcf05d315-tls-certs" pod "9c2f7980-610f-44e9-a586-b48bcf05d315" (UID: "9c2f7980-610f-44e9-a586-b48bcf05d315"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:08:21.367443 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.367418 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-dshm" pod "9c2f7980-610f-44e9-a586-b48bcf05d315" (UID: "9c2f7980-610f-44e9-a586-b48bcf05d315"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:21.368600 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.368573 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2f7980-610f-44e9-a586-b48bcf05d315-kube-api-access-2qp8h" pod "9c2f7980-610f-44e9-a586-b48bcf05d315" (UID: "9c2f7980-610f-44e9-a586-b48bcf05d315"). InnerVolumeSpecName "kube-api-access-2qp8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:08:21.467677 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.467630 4848 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c2f7980-610f-44e9-a586-b48bcf05d315-dshm\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:21.467677 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.467666 4848 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2f7980-610f-44e9-a586-b48bcf05d315-tls-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:21.467677 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.467681 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qp8h\" (UniqueName: \"kubernetes.io/projected/9c2f7980-610f-44e9-a586-b48bcf05d315-kube-api-access-2qp8h\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:21.729818 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.729783 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565"] Oct 30 21:08:21.730562 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.730537 4848 cpu_manager.go:414] "containerMap: removing container" podUID="9c2f7980-610f-44e9-a586-b48bcf05d315" containerName="main" Oct 30 21:08:21.730562 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.730555 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c2f7980-610f-44e9-a586-b48bcf05d315" containerName="main" Oct 30 21:08:21.730805 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.730676 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c2f7980-610f-44e9-a586-b48bcf05d315" containerName="main" Oct 30 21:08:21.733375 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.733352 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.740399 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.740377 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e07442-openshift-default-dockercfg-zxmtx\"" Oct 30 21:08:21.757070 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.757039 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565"] Oct 30 21:08:21.772780 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.772730 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfjwn\" (UniqueName: \"kubernetes.io/projected/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-kube-api-access-bfjwn\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.772953 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.772801 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-workload-certs\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.772953 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.772918 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-podinfo\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.773052 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.772977 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istiod-ca-cert\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.773141 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.773122 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-envoy\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.773188 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.773161 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-credential-socket\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.773251 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.773230 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-workload-socket\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.773403 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.773385 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-token\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.773484 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.773419 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-data\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.916789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.899691 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-token\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.916789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.899752 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-data\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.916789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.899864 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfjwn\" (UniqueName: \"kubernetes.io/projected/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-kube-api-access-bfjwn\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.916789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.899898 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-workload-certs\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.916789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.899984 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-podinfo\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.916789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.900048 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istiod-ca-cert\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.916789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.900193 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-envoy\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.916789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.900232 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-credential-socket\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.916789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.900257 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-workload-socket\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.926787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.918184 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-workload-certs\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.926787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.918839 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-data\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.926787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.919683 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istiod-ca-cert\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.926787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.920631 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-credential-socket\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.926787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.920832 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-workload-socket\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.957790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.952429 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-podinfo\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.957790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.955283 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfjwn\" (UniqueName: \"kubernetes.io/projected/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-kube-api-access-bfjwn\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.961791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.959111 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-envoy\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:21.980916 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:21.978265 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-token\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:22.072942 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:22.072805 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:22.143332 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:22.143298 4848 generic.go:356] "Generic (PLEG): container finished" podID="9c2f7980-610f-44e9-a586-b48bcf05d315" containerID="d858179ca4afb292f2c7084d75a06f3105ba9f7ea6cc98f700a3333beb154318" exitCode=0 Oct 30 21:08:22.143583 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:22.143563 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" event={"ID":"9c2f7980-610f-44e9-a586-b48bcf05d315","Type":"ContainerDied","Data":"d858179ca4afb292f2c7084d75a06f3105ba9f7ea6cc98f700a3333beb154318"} Oct 30 21:08:22.143658 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:22.143609 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" event={"ID":"9c2f7980-610f-44e9-a586-b48bcf05d315","Type":"ContainerDied","Data":"35d570ed30ea754aac750525bf3442234166b2185225e75f253024c9531c7498"} Oct 30 21:08:22.143658 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:22.143636 4848 scope.go:122] "RemoveContainer" containerID="d858179ca4afb292f2c7084d75a06f3105ba9f7ea6cc98f700a3333beb154318" Oct 30 21:08:22.145992 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:22.145965 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv" Oct 30 21:08:22.170462 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:22.170436 4848 scope.go:122] "RemoveContainer" containerID="d858179ca4afb292f2c7084d75a06f3105ba9f7ea6cc98f700a3333beb154318" Oct 30 21:08:22.170875 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:08:22.170854 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d858179ca4afb292f2c7084d75a06f3105ba9f7ea6cc98f700a3333beb154318\": container with ID starting with d858179ca4afb292f2c7084d75a06f3105ba9f7ea6cc98f700a3333beb154318 not found: ID does not exist" containerID="d858179ca4afb292f2c7084d75a06f3105ba9f7ea6cc98f700a3333beb154318" Oct 30 21:08:22.170926 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:22.170888 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d858179ca4afb292f2c7084d75a06f3105ba9f7ea6cc98f700a3333beb154318"} err="failed to get container status \"d858179ca4afb292f2c7084d75a06f3105ba9f7ea6cc98f700a3333beb154318\": rpc error: code = NotFound desc = could not find container \"d858179ca4afb292f2c7084d75a06f3105ba9f7ea6cc98f700a3333beb154318\": container with ID starting with d858179ca4afb292f2c7084d75a06f3105ba9f7ea6cc98f700a3333beb154318 not found: ID does not exist" Oct 30 21:08:22.178753 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:22.178726 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv"] Oct 30 21:08:22.182400 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:22.182370 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-3a65ec/auth-test-model-3a65ec-kserve-6cb4b5fb5-w4rwv"] Oct 30 21:08:22.458810 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:22.458752 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565"] Oct 30 21:08:22.475302 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:22.474155 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:08:22.475302 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:22.474303 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:08:22.475302 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:22.474355 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:08:22.526009 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:08:22.525978 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:08:22.940695 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:22.940618 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c2f7980-610f-44e9-a586-b48bcf05d315" path="/var/lib/kubelet/pods/9c2f7980-610f-44e9-a586-b48bcf05d315/volumes" Oct 30 21:08:23.152321 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:23.152281 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" event={"ID":"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8","Type":"ContainerStarted","Data":"a913616692ba833bd5b763de36ee7a8958aa5799d71eac70485a1d403c55999e"} Oct 30 21:08:23.152712 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:23.152371 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" event={"ID":"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8","Type":"ContainerStarted","Data":"3e7b31ee45a1e4208fd9e29616bea23fb1afdb3e9e33ec30f44c664d2015bf90"} Oct 30 21:08:23.187099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:23.187043 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" podStartSLOduration=2.187028027 podStartE2EDuration="2.187028027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:23.184291245 +0000 UTC m=+1808.364187752" watchObservedRunningTime="2026-10-30 21:08:23.187028027 +0000 UTC m=+1808.366924533" Oct 30 21:08:24.074095 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:24.074065 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:24.079620 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:24.079592 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:24.159738 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:24.159705 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:24.160871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:24.160851 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:08:25.776515 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:25.776482 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p"] Oct 30 21:08:25.777082 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:25.776710 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" podUID="cf35d310-f121-41d1-844c-e2bef5d31d15" containerName="istio-proxy" containerID="cri-o://3247c5dd4fde2db7467047901b3ddb58bfe08d0fe715033b69cae9ef2c516f17" gracePeriod=30 Oct 30 21:08:26.718980 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:26.718946 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj"] Oct 30 21:08:26.722678 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:26.722660 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj" Oct 30 21:08:26.727958 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:26.727936 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-e07442\"" Oct 30 21:08:26.728086 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:26.727970 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-e07442-dockercfg-xmpqq\"" Oct 30 21:08:26.735712 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:26.735688 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj"] Oct 30 21:08:26.771871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:26.771834 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wm57\" (UniqueName: \"kubernetes.io/projected/e2400054-2c0b-4ade-874c-e752a9067b54-kube-api-access-2wm57\") pod \"payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj\" (UID: \"e2400054-2c0b-4ade-874c-e752a9067b54\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj" Oct 30 21:08:26.772208 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:26.772158 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2400054-2c0b-4ade-874c-e752a9067b54-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj\" (UID: \"e2400054-2c0b-4ade-874c-e752a9067b54\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj" Oct 30 21:08:26.892787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:26.874890 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wm57\" (UniqueName: \"kubernetes.io/projected/e2400054-2c0b-4ade-874c-e752a9067b54-kube-api-access-2wm57\") pod \"payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj\" (UID: \"e2400054-2c0b-4ade-874c-e752a9067b54\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj" Oct 30 21:08:26.893524 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:26.893495 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2400054-2c0b-4ade-874c-e752a9067b54-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj\" (UID: \"e2400054-2c0b-4ade-874c-e752a9067b54\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj" Oct 30 21:08:26.924820 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:26.894800 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2400054-2c0b-4ade-874c-e752a9067b54-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj\" (UID: \"e2400054-2c0b-4ade-874c-e752a9067b54\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj" Oct 30 21:08:27.139708 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:27.126604 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wm57\" (UniqueName: \"kubernetes.io/projected/e2400054-2c0b-4ade-874c-e752a9067b54-kube-api-access-2wm57\") pod \"payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj\" (UID: \"e2400054-2c0b-4ade-874c-e752a9067b54\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj" Oct 30 21:08:27.337956 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:27.337921 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj" Oct 30 21:08:27.658948 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:27.658916 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj"] Oct 30 21:08:27.660241 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:08:27.660216 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2400054_2c0b_4ade_874c_e752a9067b54.slice/crio-819e141b55a94e5a01bca602ef7a2fe22a9eab518b498e27b86367618e3925cd WatchSource:0}: Error finding container 819e141b55a94e5a01bca602ef7a2fe22a9eab518b498e27b86367618e3925cd: Status 404 returned error can't find the container with id 819e141b55a94e5a01bca602ef7a2fe22a9eab518b498e27b86367618e3925cd Oct 30 21:08:28.189142 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:28.189109 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj" event={"ID":"e2400054-2c0b-4ade-874c-e752a9067b54","Type":"ContainerStarted","Data":"b5629ca2fe118e92a693396af5c0f61dbfadccf2229a846f82ed9a8a109eae52"} Oct 30 21:08:28.189530 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:28.189168 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj" event={"ID":"e2400054-2c0b-4ade-874c-e752a9067b54","Type":"ContainerStarted","Data":"819e141b55a94e5a01bca602ef7a2fe22a9eab518b498e27b86367618e3925cd"} Oct 30 21:08:28.189530 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:28.189199 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj" Oct 30 21:08:28.208244 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:28.208176 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj" podStartSLOduration=2.208163254 podStartE2EDuration="2.208163254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:28.206386821 +0000 UTC m=+1813.386283334" watchObservedRunningTime="2026-10-30 21:08:28.208163254 +0000 UTC m=+1813.388059761" Oct 30 21:08:31.137268 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.137242 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh" Oct 30 21:08:31.151787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.151747 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:08:31.258419 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.255456 4848 generic.go:356] "Generic (PLEG): container finished" podID="cf35d310-f121-41d1-844c-e2bef5d31d15" containerID="3247c5dd4fde2db7467047901b3ddb58bfe08d0fe715033b69cae9ef2c516f17" exitCode=0 Oct 30 21:08:31.258419 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.255720 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" event={"ID":"cf35d310-f121-41d1-844c-e2bef5d31d15","Type":"ContainerDied","Data":"3247c5dd4fde2db7467047901b3ddb58bfe08d0fe715033b69cae9ef2c516f17"} Oct 30 21:08:31.258419 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.255787 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" event={"ID":"cf35d310-f121-41d1-844c-e2bef5d31d15","Type":"ContainerDied","Data":"9e475baa125d30b33eb65e5298866650955e0e47993ddba83312d1dc90e6fb7d"} Oct 30 21:08:31.258419 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.255820 4848 scope.go:122] "RemoveContainer" containerID="3247c5dd4fde2db7467047901b3ddb58bfe08d0fe715033b69cae9ef2c516f17" Oct 30 21:08:31.258419 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.255982 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p" Oct 30 21:08:31.258419 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.256956 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf35d310-f121-41d1-844c-e2bef5d31d15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf35d310-f121-41d1-844c-e2bef5d31d15-istiod-ca-cert\") pod \"cf35d310-f121-41d1-844c-e2bef5d31d15\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " Oct 30 21:08:31.258419 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.257036 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-envoy\") pod \"cf35d310-f121-41d1-844c-e2bef5d31d15\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " Oct 30 21:08:31.258419 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.257076 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-podinfo\") pod \"cf35d310-f121-41d1-844c-e2bef5d31d15\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " Oct 30 21:08:31.258419 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.257112 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-data\") pod \"cf35d310-f121-41d1-844c-e2bef5d31d15\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " Oct 30 21:08:31.258419 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.257161 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-workload-certs\") pod \"cf35d310-f121-41d1-844c-e2bef5d31d15\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " Oct 30 21:08:31.258419 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.257240 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-credential-socket\") pod \"cf35d310-f121-41d1-844c-e2bef5d31d15\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " Oct 30 21:08:31.258419 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.257276 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-token\" (UniqueName: \"kubernetes.io/projected/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-token\") pod \"cf35d310-f121-41d1-844c-e2bef5d31d15\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " Oct 30 21:08:31.258419 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.257353 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf35d310-f121-41d1-844c-e2bef5d31d15-kube-api-access-z6vq9\" (UniqueName: \"kubernetes.io/projected/cf35d310-f121-41d1-844c-e2bef5d31d15-kube-api-access-z6vq9\") pod \"cf35d310-f121-41d1-844c-e2bef5d31d15\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " Oct 30 21:08:31.258419 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.257459 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-workload-socket\") pod \"cf35d310-f121-41d1-844c-e2bef5d31d15\" (UID: \"cf35d310-f121-41d1-844c-e2bef5d31d15\") " Oct 30 21:08:31.258419 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.257653 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-data" pod "cf35d310-f121-41d1-844c-e2bef5d31d15" (UID: "cf35d310-f121-41d1-844c-e2bef5d31d15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:31.258419 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.257922 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf35d310-f121-41d1-844c-e2bef5d31d15-istiod-ca-cert" pod "cf35d310-f121-41d1-844c-e2bef5d31d15" (UID: "cf35d310-f121-41d1-844c-e2bef5d31d15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:08:31.258419 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.258186 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf35d310-f121-41d1-844c-e2bef5d31d15-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:31.258419 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.258198 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:31.259541 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.259319 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-workload-socket" pod "cf35d310-f121-41d1-844c-e2bef5d31d15" (UID: "cf35d310-f121-41d1-844c-e2bef5d31d15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:31.260564 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.260054 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-credential-socket" pod "cf35d310-f121-41d1-844c-e2bef5d31d15" (UID: "cf35d310-f121-41d1-844c-e2bef5d31d15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:31.260564 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.260256 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-workload-certs" pod "cf35d310-f121-41d1-844c-e2bef5d31d15" (UID: "cf35d310-f121-41d1-844c-e2bef5d31d15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:31.262170 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.262135 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf35d310-f121-41d1-844c-e2bef5d31d15-kube-api-access-z6vq9" pod "cf35d310-f121-41d1-844c-e2bef5d31d15" (UID: "cf35d310-f121-41d1-844c-e2bef5d31d15"). InnerVolumeSpecName "kube-api-access-z6vq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:08:31.264320 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.264289 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-podinfo" pod "cf35d310-f121-41d1-844c-e2bef5d31d15" (UID: "cf35d310-f121-41d1-844c-e2bef5d31d15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:08:31.264420 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.264317 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-envoy" pod "cf35d310-f121-41d1-844c-e2bef5d31d15" (UID: "cf35d310-f121-41d1-844c-e2bef5d31d15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:31.296374 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.296151 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-token" pod "cf35d310-f121-41d1-844c-e2bef5d31d15" (UID: "cf35d310-f121-41d1-844c-e2bef5d31d15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:08:31.352134 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.351896 4848 scope.go:122] "RemoveContainer" containerID="3247c5dd4fde2db7467047901b3ddb58bfe08d0fe715033b69cae9ef2c516f17" Oct 30 21:08:31.352359 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:08:31.352312 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3247c5dd4fde2db7467047901b3ddb58bfe08d0fe715033b69cae9ef2c516f17\": container with ID starting with 3247c5dd4fde2db7467047901b3ddb58bfe08d0fe715033b69cae9ef2c516f17 not found: ID does not exist" containerID="3247c5dd4fde2db7467047901b3ddb58bfe08d0fe715033b69cae9ef2c516f17" Oct 30 21:08:31.352453 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.352368 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3247c5dd4fde2db7467047901b3ddb58bfe08d0fe715033b69cae9ef2c516f17"} err="failed to get container status \"3247c5dd4fde2db7467047901b3ddb58bfe08d0fe715033b69cae9ef2c516f17\": rpc error: code = NotFound desc = could not find container \"3247c5dd4fde2db7467047901b3ddb58bfe08d0fe715033b69cae9ef2c516f17\": container with ID starting with 3247c5dd4fde2db7467047901b3ddb58bfe08d0fe715033b69cae9ef2c516f17 not found: ID does not exist" Oct 30 21:08:31.359548 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.359501 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:31.359548 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.359527 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:31.359548 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.359536 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:31.359548 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.359545 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:31.362188 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.359555 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf35d310-f121-41d1-844c-e2bef5d31d15-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:31.362188 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.359566 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6vq9\" (UniqueName: \"kubernetes.io/projected/cf35d310-f121-41d1-844c-e2bef5d31d15-kube-api-access-z6vq9\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:31.362188 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.359575 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf35d310-f121-41d1-844c-e2bef5d31d15-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:31.609843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.588230 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p"] Oct 30 21:08:31.676789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:31.672533 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dg59p"] Oct 30 21:08:32.939174 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:32.939137 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf35d310-f121-41d1-844c-e2bef5d31d15" path="/var/lib/kubelet/pods/cf35d310-f121-41d1-844c-e2bef5d31d15/volumes" Oct 30 21:08:34.085046 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:34.085013 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs"] Oct 30 21:08:34.085533 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:34.085284 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" podUID="eae7f986-ff45-410c-8301-3e23132e0bc5" containerName="istio-proxy" containerID="cri-o://fd67c5c18b6a66d0192249b6f11979b2a03cdb79e667c70b9875ab4a7f616b76" gracePeriod=30 Oct 30 21:08:36.641794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.639257 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq"] Oct 30 21:08:36.641794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.640410 4848 cpu_manager.go:414] "containerMap: removing container" podUID="cf35d310-f121-41d1-844c-e2bef5d31d15" containerName="istio-proxy" Oct 30 21:08:36.641794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.640428 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf35d310-f121-41d1-844c-e2bef5d31d15" containerName="istio-proxy" Oct 30 21:08:36.641794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.640656 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf35d310-f121-41d1-844c-e2bef5d31d15" containerName="istio-proxy" Oct 30 21:08:36.647463 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.644477 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.662354 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.662326 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-6741b1-openshift-default-dockercfg-d9psc\"" Oct 30 21:08:36.668506 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.668479 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq"] Oct 30 21:08:36.801629 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.801577 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjvxp\" (UniqueName: \"kubernetes.io/projected/a1bfc658-f4c9-4280-91e9-5081b0f9416e-kube-api-access-mjvxp\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.801629 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.801631 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-workload-certs\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.801899 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.801737 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-token\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.801899 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.801777 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istiod-ca-cert\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.801899 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.801833 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-envoy\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.801899 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.801893 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-credential-socket\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.802196 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.802038 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-data\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.802196 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.802089 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-podinfo\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.820588 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.820079 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-workload-socket\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.930798 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.930625 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-workload-certs\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.930798 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.930699 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-token\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.930798 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.930723 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istiod-ca-cert\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.930798 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.930747 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-envoy\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.931174 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.930815 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-credential-socket\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.931174 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.930930 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-data\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.931174 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.930977 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-podinfo\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.931174 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.931019 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-workload-socket\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.931174 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.931148 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvxp\" (UniqueName: \"kubernetes.io/projected/a1bfc658-f4c9-4280-91e9-5081b0f9416e-kube-api-access-mjvxp\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.934786 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.931913 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-workload-certs\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.947251 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.947176 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-data\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.955785 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.949141 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istiod-ca-cert\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.955785 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.951310 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-credential-socket\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.955785 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.954493 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-workload-socket\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.984794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.981343 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-podinfo\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:36.991087 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:36.990968 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjvxp\" (UniqueName: \"kubernetes.io/projected/a1bfc658-f4c9-4280-91e9-5081b0f9416e-kube-api-access-mjvxp\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:37.008426 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:37.007136 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-envoy\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:37.008426 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:37.008368 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-token\") pod \"e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:37.296561 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:37.296229 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:37.651358 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:37.651331 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq"] Oct 30 21:08:37.655314 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:08:37.655269 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1bfc658_f4c9_4280_91e9_5081b0f9416e.slice/crio-7148562e48da5cbdd75459f0dfbf572fecdf3bc064b46b11f0c6178da829b6a1 WatchSource:0}: Error finding container 7148562e48da5cbdd75459f0dfbf572fecdf3bc064b46b11f0c6178da829b6a1: Status 404 returned error can't find the container with id 7148562e48da5cbdd75459f0dfbf572fecdf3bc064b46b11f0c6178da829b6a1 Oct 30 21:08:37.658496 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:37.658457 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:08:37.658584 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:37.658549 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:08:37.658626 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:37.658590 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:08:38.370645 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:38.370607 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" event={"ID":"a1bfc658-f4c9-4280-91e9-5081b0f9416e","Type":"ContainerStarted","Data":"f2f691e82ac81003b1527be23474f45403729a2b5c5d1e4512e21e8d22b4f51e"} Oct 30 21:08:38.370852 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:38.370673 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" event={"ID":"a1bfc658-f4c9-4280-91e9-5081b0f9416e","Type":"ContainerStarted","Data":"7148562e48da5cbdd75459f0dfbf572fecdf3bc064b46b11f0c6178da829b6a1"} Oct 30 21:08:38.391648 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:38.391593 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" podStartSLOduration=2.391579736 podStartE2EDuration="2.391579736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:38.389464357 +0000 UTC m=+1823.569360863" watchObservedRunningTime="2026-10-30 21:08:38.391579736 +0000 UTC m=+1823.571476243" Oct 30 21:08:39.200587 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.200558 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj" Oct 30 21:08:39.297804 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.297370 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:39.304822 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.304759 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:39.385350 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.385318 4848 generic.go:356] "Generic (PLEG): container finished" podID="eae7f986-ff45-410c-8301-3e23132e0bc5" containerID="fd67c5c18b6a66d0192249b6f11979b2a03cdb79e667c70b9875ab4a7f616b76" exitCode=0 Oct 30 21:08:39.385525 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.385352 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" event={"ID":"eae7f986-ff45-410c-8301-3e23132e0bc5","Type":"ContainerDied","Data":"fd67c5c18b6a66d0192249b6f11979b2a03cdb79e667c70b9875ab4a7f616b76"} Oct 30 21:08:39.385525 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.385428 4848 scope.go:122] "RemoveContainer" containerID="2d70a49c564e94d908b5e3f67166c4cf650bd120f2cb1960410f0e6500def992" Oct 30 21:08:39.385738 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.385714 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:39.386808 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.386758 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:08:39.435280 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.435247 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:08:39.504543 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.504504 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae7f986-ff45-410c-8301-3e23132e0bc5-kube-api-access-ht7jg\" (UniqueName: \"kubernetes.io/projected/eae7f986-ff45-410c-8301-3e23132e0bc5-kube-api-access-ht7jg\") pod \"eae7f986-ff45-410c-8301-3e23132e0bc5\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " Oct 30 21:08:39.504758 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.504566 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-data\") pod \"eae7f986-ff45-410c-8301-3e23132e0bc5\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " Oct 30 21:08:39.504758 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.504637 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-podinfo\") pod \"eae7f986-ff45-410c-8301-3e23132e0bc5\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " Oct 30 21:08:39.504758 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.504698 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-credential-socket\") pod \"eae7f986-ff45-410c-8301-3e23132e0bc5\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " Oct 30 21:08:39.504758 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.504743 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-workload-socket\") pod \"eae7f986-ff45-410c-8301-3e23132e0bc5\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " Oct 30 21:08:39.505081 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.504852 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-token\" (UniqueName: \"kubernetes.io/projected/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-token\") pod \"eae7f986-ff45-410c-8301-3e23132e0bc5\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " Oct 30 21:08:39.505081 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.504898 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-envoy\") pod \"eae7f986-ff45-410c-8301-3e23132e0bc5\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " Oct 30 21:08:39.505203 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.505095 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eae7f986-ff45-410c-8301-3e23132e0bc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae7f986-ff45-410c-8301-3e23132e0bc5-istiod-ca-cert\") pod \"eae7f986-ff45-410c-8301-3e23132e0bc5\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " Oct 30 21:08:39.505203 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.505189 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-workload-certs\") pod \"eae7f986-ff45-410c-8301-3e23132e0bc5\" (UID: \"eae7f986-ff45-410c-8301-3e23132e0bc5\") " Oct 30 21:08:39.507353 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.507103 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae7f986-ff45-410c-8301-3e23132e0bc5-istiod-ca-cert" pod "eae7f986-ff45-410c-8301-3e23132e0bc5" (UID: "eae7f986-ff45-410c-8301-3e23132e0bc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:08:39.507485 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.507383 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-data" pod "eae7f986-ff45-410c-8301-3e23132e0bc5" (UID: "eae7f986-ff45-410c-8301-3e23132e0bc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:39.507825 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.507617 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-workload-certs" pod "eae7f986-ff45-410c-8301-3e23132e0bc5" (UID: "eae7f986-ff45-410c-8301-3e23132e0bc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:39.508292 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.507965 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-workload-socket" pod "eae7f986-ff45-410c-8301-3e23132e0bc5" (UID: "eae7f986-ff45-410c-8301-3e23132e0bc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:39.508391 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.508295 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-credential-socket" pod "eae7f986-ff45-410c-8301-3e23132e0bc5" (UID: "eae7f986-ff45-410c-8301-3e23132e0bc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:39.509908 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.509876 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae7f986-ff45-410c-8301-3e23132e0bc5-kube-api-access-ht7jg" pod "eae7f986-ff45-410c-8301-3e23132e0bc5" (UID: "eae7f986-ff45-410c-8301-3e23132e0bc5"). InnerVolumeSpecName "kube-api-access-ht7jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:08:39.510863 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.510839 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-token" pod "eae7f986-ff45-410c-8301-3e23132e0bc5" (UID: "eae7f986-ff45-410c-8301-3e23132e0bc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:08:39.528169 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.528105 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-envoy" pod "eae7f986-ff45-410c-8301-3e23132e0bc5" (UID: "eae7f986-ff45-410c-8301-3e23132e0bc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:08:39.528590 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.528549 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-podinfo" pod "eae7f986-ff45-410c-8301-3e23132e0bc5" (UID: "eae7f986-ff45-410c-8301-3e23132e0bc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:08:39.606936 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.606903 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:39.606936 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.606934 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:39.607171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.606947 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:39.607171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.606963 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:39.607171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.606975 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:39.607171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.606986 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae7f986-ff45-410c-8301-3e23132e0bc5-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:39.607171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.607033 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:39.607171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.607046 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht7jg\" (UniqueName: \"kubernetes.io/projected/eae7f986-ff45-410c-8301-3e23132e0bc5-kube-api-access-ht7jg\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:39.607171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:39.607061 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae7f986-ff45-410c-8301-3e23132e0bc5-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:40.395257 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:40.395177 4848 generic.go:356] "Generic (PLEG): container finished" podID="127e9b91-a76e-46f6-9970-e5dcf8a0ba78" containerID="623d607cb97ef2790866d8baa631053d6a9e8f0400dd5e4ac0d59e36cc83849f" exitCode=0 Oct 30 21:08:40.395825 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:40.395253 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" event={"ID":"127e9b91-a76e-46f6-9970-e5dcf8a0ba78","Type":"ContainerDied","Data":"623d607cb97ef2790866d8baa631053d6a9e8f0400dd5e4ac0d59e36cc83849f"} Oct 30 21:08:40.395825 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:40.395729 4848 scope.go:122] "RemoveContainer" containerID="623d607cb97ef2790866d8baa631053d6a9e8f0400dd5e4ac0d59e36cc83849f" Oct 30 21:08:40.396925 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:40.396905 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" Oct 30 21:08:40.397055 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:40.396964 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs" event={"ID":"eae7f986-ff45-410c-8301-3e23132e0bc5","Type":"ContainerDied","Data":"d11e88875f100250c55397289d5d7a7d58c7a628cd939dfb4ed28572aa21c885"} Oct 30 21:08:40.397055 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:40.397013 4848 scope.go:122] "RemoveContainer" containerID="fd67c5c18b6a66d0192249b6f11979b2a03cdb79e667c70b9875ab4a7f616b76" Oct 30 21:08:40.397825 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:40.397796 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:08:40.397942 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:40.397886 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:08:40.397942 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:40.397925 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:08:40.440826 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:40.440761 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs"] Oct 30 21:08:40.444507 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:40.444478 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-89d993-openshift-default-665f5486f7-x5hgs"] Oct 30 21:08:40.936588 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:40.936550 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eae7f986-ff45-410c-8301-3e23132e0bc5" path="/var/lib/kubelet/pods/eae7f986-ff45-410c-8301-3e23132e0bc5/volumes" Oct 30 21:08:41.420848 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:41.417122 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" event={"ID":"127e9b91-a76e-46f6-9970-e5dcf8a0ba78","Type":"ContainerStarted","Data":"15c1a27811b63dc15f776b703c45bd27ea9179445a0f1fd504225f6dc8ed51e7"} Oct 30 21:08:42.400868 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:42.400834 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:08:42.401594 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:42.401570 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 21:08:42.401654 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:42.401624 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" podUID="127e9b91-a76e-46f6-9970-e5dcf8a0ba78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 21:08:43.130825 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.127068 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:08:43.400547 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.400469 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 21:08:43.400547 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.400529 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" podUID="127e9b91-a76e-46f6-9970-e5dcf8a0ba78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 21:08:43.734864 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.734815 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg"] Oct 30 21:08:43.735998 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.735972 4848 cpu_manager.go:414] "containerMap: removing container" podUID="eae7f986-ff45-410c-8301-3e23132e0bc5" containerName="istio-proxy" Oct 30 21:08:43.735998 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.735996 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eae7f986-ff45-410c-8301-3e23132e0bc5" containerName="istio-proxy" Oct 30 21:08:43.736197 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.736051 4848 cpu_manager.go:414] "containerMap: removing container" podUID="eae7f986-ff45-410c-8301-3e23132e0bc5" containerName="istio-proxy" Oct 30 21:08:43.736197 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.736059 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eae7f986-ff45-410c-8301-3e23132e0bc5" containerName="istio-proxy" Oct 30 21:08:43.736303 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.736225 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="eae7f986-ff45-410c-8301-3e23132e0bc5" containerName="istio-proxy" Oct 30 21:08:43.742210 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.742185 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg"] Oct 30 21:08:43.742351 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.742309 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" Oct 30 21:08:43.746603 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.746580 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-e07442-serving-cert\"" Oct 30 21:08:43.839987 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.839439 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d3d52ea-ca0a-456e-bf85-896e16fedc84-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg\" (UID: \"9d3d52ea-ca0a-456e-bf85-896e16fedc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" Oct 30 21:08:43.839987 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.839577 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj8fh\" (UniqueName: \"kubernetes.io/projected/9d3d52ea-ca0a-456e-bf85-896e16fedc84-kube-api-access-zj8fh\") pod \"maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg\" (UID: \"9d3d52ea-ca0a-456e-bf85-896e16fedc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" Oct 30 21:08:43.839987 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.839739 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d3d52ea-ca0a-456e-bf85-896e16fedc84-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg\" (UID: \"9d3d52ea-ca0a-456e-bf85-896e16fedc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" Oct 30 21:08:43.839987 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.839865 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d3d52ea-ca0a-456e-bf85-896e16fedc84-metrics-certs\") pod \"maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg\" (UID: \"9d3d52ea-ca0a-456e-bf85-896e16fedc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" Oct 30 21:08:43.963344 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.963304 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj8fh\" (UniqueName: \"kubernetes.io/projected/9d3d52ea-ca0a-456e-bf85-896e16fedc84-kube-api-access-zj8fh\") pod \"maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg\" (UID: \"9d3d52ea-ca0a-456e-bf85-896e16fedc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" Oct 30 21:08:43.963748 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.963726 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d3d52ea-ca0a-456e-bf85-896e16fedc84-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg\" (UID: \"9d3d52ea-ca0a-456e-bf85-896e16fedc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" Oct 30 21:08:43.964040 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.964020 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d3d52ea-ca0a-456e-bf85-896e16fedc84-metrics-certs\") pod \"maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg\" (UID: \"9d3d52ea-ca0a-456e-bf85-896e16fedc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" Oct 30 21:08:43.964254 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.964238 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d3d52ea-ca0a-456e-bf85-896e16fedc84-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg\" (UID: \"9d3d52ea-ca0a-456e-bf85-896e16fedc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" Oct 30 21:08:43.965200 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.965181 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d3d52ea-ca0a-456e-bf85-896e16fedc84-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg\" (UID: \"9d3d52ea-ca0a-456e-bf85-896e16fedc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" Oct 30 21:08:43.967909 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.967857 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d3d52ea-ca0a-456e-bf85-896e16fedc84-metrics-certs\") pod \"maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg\" (UID: \"9d3d52ea-ca0a-456e-bf85-896e16fedc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" Oct 30 21:08:43.972172 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.972138 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d3d52ea-ca0a-456e-bf85-896e16fedc84-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg\" (UID: \"9d3d52ea-ca0a-456e-bf85-896e16fedc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" Oct 30 21:08:43.974669 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:43.974642 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj8fh\" (UniqueName: \"kubernetes.io/projected/9d3d52ea-ca0a-456e-bf85-896e16fedc84-kube-api-access-zj8fh\") pod \"maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg\" (UID: \"9d3d52ea-ca0a-456e-bf85-896e16fedc84\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" Oct 30 21:08:44.060184 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:44.059919 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" Oct 30 21:08:44.400992 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:44.400916 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 21:08:44.400992 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:44.400977 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" podUID="127e9b91-a76e-46f6-9970-e5dcf8a0ba78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 21:08:44.597898 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:44.597854 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg"] Oct 30 21:08:44.606352 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:44.606326 4848 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:08:45.040976 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:45.040945 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z"] Oct 30 21:08:45.412850 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:45.409308 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 21:08:45.412850 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:45.409368 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" podUID="127e9b91-a76e-46f6-9970-e5dcf8a0ba78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 21:08:45.590222 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:45.590186 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" event={"ID":"9d3d52ea-ca0a-456e-bf85-896e16fedc84","Type":"ContainerStarted","Data":"26716bf6484e87cb4548f364a848166c75e1c11a572662f3b122b2f3bf469280"} Oct 30 21:08:45.590407 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:45.590241 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" event={"ID":"9d3d52ea-ca0a-456e-bf85-896e16fedc84","Type":"ContainerStarted","Data":"46aee9ba2944af28e80d719179f911ccdcc922d8a68b5bd6d9adcc84df2d5146"} Oct 30 21:08:45.590589 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:45.590575 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" Oct 30 21:08:46.400320 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:46.400286 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 21:08:46.400510 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:46.400356 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" podUID="127e9b91-a76e-46f6-9970-e5dcf8a0ba78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 21:08:47.405519 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:47.405489 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:08:47.406726 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:47.406706 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:08:47.425920 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:47.425863 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" podStartSLOduration=3.8939606429999998 podStartE2EDuration="4.425845032s" totalImagesPullingTime="531.884389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:08:45.615335232 +0000 UTC m=+1830.795231740" watchObservedRunningTime="2026-10-30 21:08:47.425845032 +0000 UTC m=+1832.605741541" Oct 30 21:08:50.043239 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:50.043194 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z" podUID="8588a492-0418-481f-8a0c-913be7c9b50e" containerName="payload-pre-processing" containerID="cri-o://3ca9a2b1aa9eab5a46cee0ce37e49593e422c589b019f7e4abbab9ac48ecae20" gracePeriod=25 Oct 30 21:08:50.358722 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:50.358695 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z" Oct 30 21:08:50.453124 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:50.453093 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8588a492-0418-481f-8a0c-913be7c9b50e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8588a492-0418-481f-8a0c-913be7c9b50e-plugins-config-volume\") pod \"8588a492-0418-481f-8a0c-913be7c9b50e\" (UID: \"8588a492-0418-481f-8a0c-913be7c9b50e\") " Oct 30 21:08:50.453269 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:50.453154 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8588a492-0418-481f-8a0c-913be7c9b50e-kube-api-access-7l8px\" (UniqueName: \"kubernetes.io/projected/8588a492-0418-481f-8a0c-913be7c9b50e-kube-api-access-7l8px\") pod \"8588a492-0418-481f-8a0c-913be7c9b50e\" (UID: \"8588a492-0418-481f-8a0c-913be7c9b50e\") " Oct 30 21:08:50.453459 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:50.453438 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8588a492-0418-481f-8a0c-913be7c9b50e-plugins-config-volume" pod "8588a492-0418-481f-8a0c-913be7c9b50e" (UID: "8588a492-0418-481f-8a0c-913be7c9b50e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:08:50.453705 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:50.453692 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8588a492-0418-481f-8a0c-913be7c9b50e-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:50.455226 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:50.455204 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8588a492-0418-481f-8a0c-913be7c9b50e-kube-api-access-7l8px" pod "8588a492-0418-481f-8a0c-913be7c9b50e" (UID: "8588a492-0418-481f-8a0c-913be7c9b50e"). InnerVolumeSpecName "kube-api-access-7l8px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:08:50.555219 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:50.555185 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l8px\" (UniqueName: \"kubernetes.io/projected/8588a492-0418-481f-8a0c-913be7c9b50e-kube-api-access-7l8px\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:50.626190 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:50.626104 4848 generic.go:356] "Generic (PLEG): container finished" podID="8588a492-0418-481f-8a0c-913be7c9b50e" containerID="3ca9a2b1aa9eab5a46cee0ce37e49593e422c589b019f7e4abbab9ac48ecae20" exitCode=0 Oct 30 21:08:50.626190 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:50.626145 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z" Oct 30 21:08:50.626190 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:50.626179 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z" event={"ID":"8588a492-0418-481f-8a0c-913be7c9b50e","Type":"ContainerDied","Data":"3ca9a2b1aa9eab5a46cee0ce37e49593e422c589b019f7e4abbab9ac48ecae20"} Oct 30 21:08:50.626438 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:50.626231 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z" event={"ID":"8588a492-0418-481f-8a0c-913be7c9b50e","Type":"ContainerDied","Data":"0ea965524454df931f4796f27d7023fac46646df6e8d75d8049775ff80e17216"} Oct 30 21:08:50.626438 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:50.626255 4848 scope.go:122] "RemoveContainer" containerID="3ca9a2b1aa9eab5a46cee0ce37e49593e422c589b019f7e4abbab9ac48ecae20" Oct 30 21:08:50.637163 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:50.637136 4848 scope.go:122] "RemoveContainer" containerID="3ca9a2b1aa9eab5a46cee0ce37e49593e422c589b019f7e4abbab9ac48ecae20" Oct 30 21:08:50.637480 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:08:50.637461 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ca9a2b1aa9eab5a46cee0ce37e49593e422c589b019f7e4abbab9ac48ecae20\": container with ID starting with 3ca9a2b1aa9eab5a46cee0ce37e49593e422c589b019f7e4abbab9ac48ecae20 not found: ID does not exist" containerID="3ca9a2b1aa9eab5a46cee0ce37e49593e422c589b019f7e4abbab9ac48ecae20" Oct 30 21:08:50.637552 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:50.637493 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ca9a2b1aa9eab5a46cee0ce37e49593e422c589b019f7e4abbab9ac48ecae20"} err="failed to get container status \"3ca9a2b1aa9eab5a46cee0ce37e49593e422c589b019f7e4abbab9ac48ecae20\": rpc error: code = NotFound desc = could not find container \"3ca9a2b1aa9eab5a46cee0ce37e49593e422c589b019f7e4abbab9ac48ecae20\": container with ID starting with 3ca9a2b1aa9eab5a46cee0ce37e49593e422c589b019f7e4abbab9ac48ecae20 not found: ID does not exist" Oct 30 21:08:50.650386 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:50.650353 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z"] Oct 30 21:08:50.654423 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:50.654397 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-49c7c2-55f9895595-5mh2z"] Oct 30 21:08:50.934538 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:50.934506 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8588a492-0418-481f-8a0c-913be7c9b50e" path="/var/lib/kubelet/pods/8588a492-0418-481f-8a0c-913be7c9b50e/volumes" Oct 30 21:08:51.602792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:51.602735 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" Oct 30 21:08:52.514228 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:08:52.514186 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:08:55.265819 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:55.265759 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7"] Oct 30 21:08:55.266643 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:55.266591 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" podUID="08dc8fd5-46c4-4c83-9ae2-d01ccf85656c" containerName="maas-api" containerID="cri-o://fd14d21244cdad9656f8d07168d06be241554a1db77f3395e81f4011b3a70084" gracePeriod=30 Oct 30 21:08:55.835207 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:55.834858 4848 generic.go:356] "Generic (PLEG): container finished" podID="08dc8fd5-46c4-4c83-9ae2-d01ccf85656c" containerID="fd14d21244cdad9656f8d07168d06be241554a1db77f3395e81f4011b3a70084" exitCode=0 Oct 30 21:08:55.835207 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:55.835059 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" event={"ID":"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c","Type":"ContainerDied","Data":"fd14d21244cdad9656f8d07168d06be241554a1db77f3395e81f4011b3a70084"} Oct 30 21:08:55.852712 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:55.852681 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" Oct 30 21:08:55.954469 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:55.954440 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-kube-api-access-tfc2j\" (UniqueName: \"kubernetes.io/projected/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-kube-api-access-tfc2j\") pod \"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c\" (UID: \"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c\") " Oct 30 21:08:55.954636 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:55.954611 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-openshift-service-ca\") pod \"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c\" (UID: \"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c\") " Oct 30 21:08:55.954699 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:55.954686 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-metrics-certs\") pod \"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c\" (UID: \"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c\") " Oct 30 21:08:55.954775 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:55.954730 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-maas-api-tls\") pod \"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c\" (UID: \"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c\") " Oct 30 21:08:55.955052 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:55.955001 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-openshift-service-ca" pod "08dc8fd5-46c4-4c83-9ae2-d01ccf85656c" (UID: "08dc8fd5-46c4-4c83-9ae2-d01ccf85656c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:08:55.955499 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:55.955477 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:55.957134 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:55.957084 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-maas-api-tls" pod "08dc8fd5-46c4-4c83-9ae2-d01ccf85656c" (UID: "08dc8fd5-46c4-4c83-9ae2-d01ccf85656c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:08:55.957258 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:55.957165 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-metrics-certs" pod "08dc8fd5-46c4-4c83-9ae2-d01ccf85656c" (UID: "08dc8fd5-46c4-4c83-9ae2-d01ccf85656c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:08:55.957258 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:55.957200 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-kube-api-access-tfc2j" pod "08dc8fd5-46c4-4c83-9ae2-d01ccf85656c" (UID: "08dc8fd5-46c4-4c83-9ae2-d01ccf85656c"). InnerVolumeSpecName "kube-api-access-tfc2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:08:56.056630 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:56.056596 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:56.056630 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:56.056624 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:56.056630 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:56.056634 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfc2j\" (UniqueName: \"kubernetes.io/projected/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c-kube-api-access-tfc2j\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:08:56.883923 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:56.882645 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" event={"ID":"08dc8fd5-46c4-4c83-9ae2-d01ccf85656c","Type":"ContainerDied","Data":"fc918358033edfed4118a9f1efb38cea44a27052b2280fb9fb3b04568473adad"} Oct 30 21:08:56.883923 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:56.882714 4848 scope.go:122] "RemoveContainer" containerID="fd14d21244cdad9656f8d07168d06be241554a1db77f3395e81f4011b3a70084" Oct 30 21:08:56.883923 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:56.882928 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" Oct 30 21:08:56.911318 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:56.911291 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7"] Oct 30 21:08:56.917881 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:56.917858 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7"] Oct 30 21:08:56.934009 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:56.933981 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08dc8fd5-46c4-4c83-9ae2-d01ccf85656c" path="/var/lib/kubelet/pods/08dc8fd5-46c4-4c83-9ae2-d01ccf85656c/volumes" Oct 30 21:08:58.547143 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:08:58.547104 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-49c7c2-64b556788c-nn7v7" podUID="08dc8fd5-46c4-4c83-9ae2-d01ccf85656c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.204:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:09:04.269756 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.269718 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k"] Oct 30 21:09:04.270829 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.270807 4848 cpu_manager.go:414] "containerMap: removing container" podUID="8588a492-0418-481f-8a0c-913be7c9b50e" containerName="payload-pre-processing" Oct 30 21:09:04.270880 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.270831 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8588a492-0418-481f-8a0c-913be7c9b50e" containerName="payload-pre-processing" Oct 30 21:09:04.270880 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.270861 4848 cpu_manager.go:414] "containerMap: removing container" podUID="08dc8fd5-46c4-4c83-9ae2-d01ccf85656c" containerName="maas-api" Oct 30 21:09:04.270880 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.270870 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08dc8fd5-46c4-4c83-9ae2-d01ccf85656c" containerName="maas-api" Oct 30 21:09:04.271106 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.271093 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="eae7f986-ff45-410c-8301-3e23132e0bc5" containerName="istio-proxy" Oct 30 21:09:04.271154 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.271125 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="8588a492-0418-481f-8a0c-913be7c9b50e" containerName="payload-pre-processing" Oct 30 21:09:04.271154 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.271139 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="08dc8fd5-46c4-4c83-9ae2-d01ccf85656c" containerName="maas-api" Oct 30 21:09:04.277232 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.277207 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.291973 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.291943 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k"] Oct 30 21:09:04.294645 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.294617 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-7386b4-openshift-default-dockercfg-pcltc\"" Oct 30 21:09:04.380372 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.380326 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-token\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.380531 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.380427 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-workload-socket\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.380531 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.380460 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-credential-socket\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.380531 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.380523 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-data\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.380683 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.380566 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwp99\" (UniqueName: \"kubernetes.io/projected/9f01e20c-1755-4c64-9dab-f397c8fc4852-kube-api-access-wwp99\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.380683 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.380611 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f01e20c-1755-4c64-9dab-f397c8fc4852-istiod-ca-cert\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.380683 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.380660 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-podinfo\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.380801 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.380714 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-workload-certs\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.380894 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.380878 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-envoy\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.500794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.493855 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-envoy\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.500794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.493943 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-token\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.500794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.494021 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-workload-socket\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.500794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.494056 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-credential-socket\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.500794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.494125 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-data\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.500794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.494173 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwp99\" (UniqueName: \"kubernetes.io/projected/9f01e20c-1755-4c64-9dab-f397c8fc4852-kube-api-access-wwp99\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.500794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.494225 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f01e20c-1755-4c64-9dab-f397c8fc4852-istiod-ca-cert\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.500794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.494278 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-podinfo\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.500794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.494340 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-workload-certs\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.500794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.494898 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-workload-certs\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.528852 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.525808 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-workload-socket\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.528852 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.526109 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-credential-socket\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.528852 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.526366 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-data\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.528852 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.527441 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f01e20c-1755-4c64-9dab-f397c8fc4852-istiod-ca-cert\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.543790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.533884 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-envoy\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.568792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.553663 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwp99\" (UniqueName: \"kubernetes.io/projected/9f01e20c-1755-4c64-9dab-f397c8fc4852-kube-api-access-wwp99\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.597790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.593745 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-token\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.612794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.608724 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-podinfo\") pod \"e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:04.678234 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:04.676781 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:05.094200 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:05.094167 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k"] Oct 30 21:09:05.094977 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:09:05.094946 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f01e20c_1755_4c64_9dab_f397c8fc4852.slice/crio-7e1d2a626dd2e8a2c4fb203bb5d998f83df28766f400ae6be7a03348a490e855 WatchSource:0}: Error finding container 7e1d2a626dd2e8a2c4fb203bb5d998f83df28766f400ae6be7a03348a490e855: Status 404 returned error can't find the container with id 7e1d2a626dd2e8a2c4fb203bb5d998f83df28766f400ae6be7a03348a490e855 Oct 30 21:09:05.098193 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:05.098158 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:05.098302 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:05.098227 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:05.098302 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:05.098262 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:06.092025 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:06.091980 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" event={"ID":"9f01e20c-1755-4c64-9dab-f397c8fc4852","Type":"ContainerStarted","Data":"b2fbdcbd5e87f2aed825dbbac88b44aeaca17be0f266d81918ed9c5f6219d60e"} Oct 30 21:09:06.092467 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:06.092056 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" event={"ID":"9f01e20c-1755-4c64-9dab-f397c8fc4852","Type":"ContainerStarted","Data":"7e1d2a626dd2e8a2c4fb203bb5d998f83df28766f400ae6be7a03348a490e855"} Oct 30 21:09:06.153011 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:06.152865 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" podStartSLOduration=2.152840932 podStartE2EDuration="2.152840932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:06.113054993 +0000 UTC m=+1851.292951510" watchObservedRunningTime="2026-10-30 21:09:06.152840932 +0000 UTC m=+1851.332737440" Oct 30 21:09:06.677593 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:06.677556 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:06.685727 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:06.685695 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:06.840390 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:06.840207 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb"] Oct 30 21:09:06.855795 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:06.855024 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:06.906418 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:06.906377 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb"] Oct 30 21:09:06.974375 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:06.974285 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-token\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:06.974375 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:06.974351 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czqsj\" (UniqueName: \"kubernetes.io/projected/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-kube-api-access-czqsj\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:06.974632 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:06.974605 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istiod-ca-cert\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:06.974689 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:06.974657 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-data\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:06.974795 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:06.974776 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-envoy\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:06.974895 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:06.974880 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-credential-socket\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:06.974960 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:06.974912 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-workload-certs\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:06.975043 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:06.975030 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-workload-socket\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:06.975102 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:06.975081 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-podinfo\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:07.076577 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.076539 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-workload-socket\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:07.076760 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.076592 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-podinfo\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:07.076760 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.076638 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-token\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:07.076760 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.076664 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czqsj\" (UniqueName: \"kubernetes.io/projected/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-kube-api-access-czqsj\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:07.076979 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.076820 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istiod-ca-cert\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:07.076979 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.076854 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-data\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:07.076979 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.076892 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-envoy\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:07.077118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.076989 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-credential-socket\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:07.077118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.077034 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-workload-certs\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:07.077118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.077042 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-workload-socket\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:07.077348 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.077324 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-data\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:07.077435 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.077348 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-credential-socket\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:07.077625 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.077600 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-workload-certs\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:07.077801 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.077627 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istiod-ca-cert\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:07.079212 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.079182 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-envoy\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:07.079321 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.079305 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-podinfo\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:07.086516 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.086483 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czqsj\" (UniqueName: \"kubernetes.io/projected/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-kube-api-access-czqsj\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:07.086593 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.086553 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-token\") pod \"e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:07.102441 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.102386 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:07.103445 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.103423 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:09:07.193131 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.193098 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:07.620840 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.619624 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb"] Oct 30 21:09:07.693830 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.692344 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:07.693830 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.692445 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:07.693830 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:07.692486 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:08.114895 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:08.114475 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" event={"ID":"91501d35-e8f3-4a20-93b6-6b2bda59e4e4","Type":"ContainerStarted","Data":"5cb33479546f22b38e72b41c8aa93977f768a806674c4c6c9b2ee8d746d33c08"} Oct 30 21:09:08.114895 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:08.114586 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" event={"ID":"91501d35-e8f3-4a20-93b6-6b2bda59e4e4","Type":"ContainerStarted","Data":"4d8b8bd23e6d3d853cd7c2b9840c45120d1388cc7052604770c6a6af935afafa"} Oct 30 21:09:08.153244 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:08.153134 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" podStartSLOduration=2.153113839 podStartE2EDuration="2.153113839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:08.145006968 +0000 UTC m=+1853.324903476" watchObservedRunningTime="2026-10-30 21:09:08.153113839 +0000 UTC m=+1853.333010347" Oct 30 21:09:08.193919 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:08.193882 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:08.195841 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:08.195818 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 30 21:09:08.195990 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:08.195863 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" podUID="91501d35-e8f3-4a20-93b6-6b2bda59e4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 30 21:09:09.194409 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:09.194373 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 30 21:09:09.194857 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:09.194429 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" podUID="91501d35-e8f3-4a20-93b6-6b2bda59e4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 30 21:09:10.193561 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:10.193527 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 30 21:09:11.149052 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:10.193581 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" podUID="91501d35-e8f3-4a20-93b6-6b2bda59e4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 30 21:09:11.149052 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:10.201883 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:09:11.149052 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:10.201923 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" podUID="f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:09:11.193984 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:11.193952 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 30 21:09:11.194183 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:11.194011 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" podUID="91501d35-e8f3-4a20-93b6-6b2bda59e4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 30 21:09:12.167154 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:12.167122 4848 generic.go:356] "Generic (PLEG): container finished" podID="3402ddb4-1b5f-4458-a868-19a4add51793" containerID="63c93aaaee2974cfa22d1f350b159ccffa635f838191e2bcb40faec6dc5e7126" exitCode=0 Oct 30 21:09:12.167618 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:12.167202 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" event={"ID":"3402ddb4-1b5f-4458-a868-19a4add51793","Type":"ContainerDied","Data":"63c93aaaee2974cfa22d1f350b159ccffa635f838191e2bcb40faec6dc5e7126"} Oct 30 21:09:12.167728 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:12.167710 4848 scope.go:122] "RemoveContainer" containerID="63c93aaaee2974cfa22d1f350b159ccffa635f838191e2bcb40faec6dc5e7126" Oct 30 21:09:12.169310 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:12.169288 4848 generic.go:356] "Generic (PLEG): container finished" podID="f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" containerID="a913616692ba833bd5b763de36ee7a8958aa5799d71eac70485a1d403c55999e" exitCode=0 Oct 30 21:09:12.169411 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:12.169329 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" event={"ID":"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8","Type":"ContainerDied","Data":"a913616692ba833bd5b763de36ee7a8958aa5799d71eac70485a1d403c55999e"} Oct 30 21:09:12.169777 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:12.169742 4848 scope.go:122] "RemoveContainer" containerID="a913616692ba833bd5b763de36ee7a8958aa5799d71eac70485a1d403c55999e" Oct 30 21:09:12.169871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:12.169747 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:12.169939 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:12.169869 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:12.169939 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:12.169917 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:12.171412 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:12.171385 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:12.171493 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:12.171451 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:12.171493 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:12.171482 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:12.193568 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:12.193541 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 30 21:09:12.193656 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:12.193596 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" podUID="91501d35-e8f3-4a20-93b6-6b2bda59e4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 30 21:09:13.190750 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:13.190706 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" event={"ID":"3402ddb4-1b5f-4458-a868-19a4add51793","Type":"ContainerStarted","Data":"413f427ba2ad7bd77d8b8dfdffbf3aad21858a314cbcb70f9f2cd4352fa7648d"} Oct 30 21:09:13.193705 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:13.193675 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 30 21:09:13.194365 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:13.194335 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" podUID="91501d35-e8f3-4a20-93b6-6b2bda59e4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 30 21:09:13.196710 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:13.196684 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" event={"ID":"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8","Type":"ContainerStarted","Data":"721652c39a3bca0f03817f8f78389b303cb87368a19c36037f47fec2562ccfc4"} Oct 30 21:09:13.657144 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:13.656996 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:09:13.663610 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:13.663271 4848 patch_prober.go:29] interesting pod/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 30 21:09:13.663610 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:13.663329 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" podUID="3402ddb4-1b5f-4458-a868-19a4add51793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 30 21:09:14.075225 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:14.075189 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:09:14.075592 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:14.075568 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 30 21:09:14.075793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:14.075746 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" podUID="f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 30 21:09:14.194815 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:14.194177 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body= Oct 30 21:09:14.194815 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:14.194236 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" podUID="91501d35-e8f3-4a20-93b6-6b2bda59e4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" Oct 30 21:09:14.661935 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:14.661900 4848 patch_prober.go:29] interesting pod/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 30 21:09:14.662286 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:14.662255 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" podUID="3402ddb4-1b5f-4458-a868-19a4add51793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 30 21:09:15.074173 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:15.074090 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 30 21:09:15.074173 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:15.074153 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" podUID="f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 30 21:09:15.704500 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:15.676415 4848 patch_prober.go:29] interesting pod/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 30 21:09:15.704500 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:15.676469 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" podUID="3402ddb4-1b5f-4458-a868-19a4add51793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 30 21:09:15.957843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:15.954297 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:15.957843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:15.956479 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:15.966801 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:15.964056 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:09:16.023920 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:16.019932 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:09:16.077952 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:16.077897 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 30 21:09:16.078342 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:16.078292 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" podUID="f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 30 21:09:16.655906 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:16.655874 4848 patch_prober.go:29] interesting pod/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 30 21:09:16.656219 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:16.656182 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" podUID="3402ddb4-1b5f-4458-a868-19a4add51793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 30 21:09:17.074803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:17.074736 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 30 21:09:17.075314 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:17.074828 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" podUID="f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 30 21:09:17.658420 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:17.658313 4848 patch_prober.go:29] interesting pod/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 30 21:09:17.658420 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:17.658383 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" podUID="3402ddb4-1b5f-4458-a868-19a4add51793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 30 21:09:18.075176 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:18.075139 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Oct 30 21:09:18.075597 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:18.075202 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" podUID="f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Oct 30 21:09:18.655590 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:18.655547 4848 patch_prober.go:29] interesting pod/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 30 21:09:18.655799 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:18.655614 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" podUID="3402ddb4-1b5f-4458-a868-19a4add51793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 30 21:09:19.182862 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:19.182831 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:09:19.184110 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:19.184087 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:09:19.185474 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:19.185449 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:09:19.659555 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:19.659472 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:09:19.660544 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:19.660524 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:09:22.511823 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:09:22.511780 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:09:26.012793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:26.004214 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh"] Oct 30 21:09:30.679496 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:30.679462 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8"] Oct 30 21:09:30.683700 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:30.683676 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" Oct 30 21:09:30.693188 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:30.693157 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8"] Oct 30 21:09:30.696392 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:30.696352 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-7386b4-serving-cert\"" Oct 30 21:09:30.849888 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:30.825969 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb21377-14eb-4b8d-86dc-19107022c1c9-metrics-certs\") pod \"maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8\" (UID: \"4bb21377-14eb-4b8d-86dc-19107022c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" Oct 30 21:09:30.850260 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:30.850221 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb21377-14eb-4b8d-86dc-19107022c1c9-maas-api-tls\") pod \"maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8\" (UID: \"4bb21377-14eb-4b8d-86dc-19107022c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" Oct 30 21:09:30.850487 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:30.850464 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtdrz\" (UniqueName: \"kubernetes.io/projected/4bb21377-14eb-4b8d-86dc-19107022c1c9-kube-api-access-vtdrz\") pod \"maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8\" (UID: \"4bb21377-14eb-4b8d-86dc-19107022c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" Oct 30 21:09:30.850697 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:30.850679 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb21377-14eb-4b8d-86dc-19107022c1c9-openshift-service-ca\") pod \"maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8\" (UID: \"4bb21377-14eb-4b8d-86dc-19107022c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" Oct 30 21:09:30.974904 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:30.974811 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb21377-14eb-4b8d-86dc-19107022c1c9-metrics-certs\") pod \"maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8\" (UID: \"4bb21377-14eb-4b8d-86dc-19107022c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" Oct 30 21:09:30.975201 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:30.975182 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb21377-14eb-4b8d-86dc-19107022c1c9-maas-api-tls\") pod \"maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8\" (UID: \"4bb21377-14eb-4b8d-86dc-19107022c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" Oct 30 21:09:30.975348 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:30.975332 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtdrz\" (UniqueName: \"kubernetes.io/projected/4bb21377-14eb-4b8d-86dc-19107022c1c9-kube-api-access-vtdrz\") pod \"maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8\" (UID: \"4bb21377-14eb-4b8d-86dc-19107022c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" Oct 30 21:09:30.975514 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:30.975499 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb21377-14eb-4b8d-86dc-19107022c1c9-openshift-service-ca\") pod \"maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8\" (UID: \"4bb21377-14eb-4b8d-86dc-19107022c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" Oct 30 21:09:30.976472 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:30.976447 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb21377-14eb-4b8d-86dc-19107022c1c9-openshift-service-ca\") pod \"maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8\" (UID: \"4bb21377-14eb-4b8d-86dc-19107022c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" Oct 30 21:09:30.980641 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:30.980610 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb21377-14eb-4b8d-86dc-19107022c1c9-metrics-certs\") pod \"maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8\" (UID: \"4bb21377-14eb-4b8d-86dc-19107022c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" Oct 30 21:09:30.986125 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:30.986094 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb21377-14eb-4b8d-86dc-19107022c1c9-maas-api-tls\") pod \"maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8\" (UID: \"4bb21377-14eb-4b8d-86dc-19107022c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" Oct 30 21:09:30.990008 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:30.989983 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtdrz\" (UniqueName: \"kubernetes.io/projected/4bb21377-14eb-4b8d-86dc-19107022c1c9-kube-api-access-vtdrz\") pod \"maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8\" (UID: \"4bb21377-14eb-4b8d-86dc-19107022c1c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" Oct 30 21:09:31.009910 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:31.009844 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh" podUID="a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26" containerName="payload-pre-processing" containerID="cri-o://19fc0c17524d871cf624e0206ee9a9008076c4440603510885a4f7b433373195" gracePeriod=25 Oct 30 21:09:31.040514 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:31.040356 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" Oct 30 21:09:31.117453 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:09:31.114403 4848 logging.go:55] [core] [Channel #3014 SubChannel #3015]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.213:50052", ServerName: "10.134.0.213:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.213:50052: connect: connection refused" Oct 30 21:09:31.650952 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:31.650872 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8"] Oct 30 21:09:31.659625 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:31.659591 4848 generic.go:356] "Generic (PLEG): container finished" podID="a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26" containerID="19fc0c17524d871cf624e0206ee9a9008076c4440603510885a4f7b433373195" exitCode=0 Oct 30 21:09:31.659816 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:31.659692 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh" event={"ID":"a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26","Type":"ContainerDied","Data":"19fc0c17524d871cf624e0206ee9a9008076c4440603510885a4f7b433373195"} Oct 30 21:09:31.662137 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:31.662092 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" event={"ID":"4bb21377-14eb-4b8d-86dc-19107022c1c9","Type":"ContainerStarted","Data":"9697350553c33aa64d3b41b3960937e8570a799f1f8eb9cec9fdc815dc10297e"} Oct 30 21:09:31.700424 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:31.700388 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh" Oct 30 21:09:31.760740 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:31.760699 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26-kube-api-access-qtfq5\" (UniqueName: \"kubernetes.io/projected/a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26-kube-api-access-qtfq5\") pod \"a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26\" (UID: \"a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26\") " Oct 30 21:09:31.760953 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:31.760798 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26-plugins-config-volume\") pod \"a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26\" (UID: \"a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26\") " Oct 30 21:09:31.761643 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:31.761615 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26-plugins-config-volume" pod "a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26" (UID: "a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:09:31.763637 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:31.763606 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26-kube-api-access-qtfq5" pod "a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26" (UID: "a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26"). InnerVolumeSpecName "kube-api-access-qtfq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:09:31.862653 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:31.862617 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtfq5\" (UniqueName: \"kubernetes.io/projected/a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26-kube-api-access-qtfq5\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:09:31.862653 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:31.862650 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:09:32.125044 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:32.124998 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh" podUID="a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.213:50052\" within 1s: context deadline exceeded" Oct 30 21:09:32.125595 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:09:32.125576 4848 logging.go:55] [core] [Channel #3014 SubChannel #3015]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.213:50052", ServerName: "10.134.0.213:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.213:50052: operation was canceled" Oct 30 21:09:32.671600 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:32.671549 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh" event={"ID":"a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26","Type":"ContainerDied","Data":"01005e6c83fcfa9d7947c7c165f8c6f68947ae362dd396906a385911f6788bc4"} Oct 30 21:09:32.671826 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:32.671629 4848 scope.go:122] "RemoveContainer" containerID="19fc0c17524d871cf624e0206ee9a9008076c4440603510885a4f7b433373195" Oct 30 21:09:32.671826 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:32.671571 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh" Oct 30 21:09:32.674860 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:32.673961 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" event={"ID":"4bb21377-14eb-4b8d-86dc-19107022c1c9","Type":"ContainerStarted","Data":"576124d0ab21466d91e270a8456ff89e5b32efe09cebdb386f10aec75269700f"} Oct 30 21:09:32.674860 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:32.674059 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" Oct 30 21:09:32.752851 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:32.752791 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" podStartSLOduration=2.215348321 podStartE2EDuration="2.752755157s" totalImagesPullingTime="537.406836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:32.741082711 +0000 UTC m=+1877.920979219" watchObservedRunningTime="2026-10-30 21:09:32.752755157 +0000 UTC m=+1877.932651664" Oct 30 21:09:32.760084 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:32.760053 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh"] Oct 30 21:09:32.763536 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:32.763505 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-435414-5bf8cdfd95-brtbh"] Oct 30 21:09:32.952249 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:32.952212 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26" path="/var/lib/kubelet/pods/a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26/volumes" Oct 30 21:09:38.698046 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:38.698018 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" Oct 30 21:09:40.097791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:40.097623 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": read tcp 10.134.0.2:48860->10.134.0.216:15021: read: connection reset by peer" start-of-body= Oct 30 21:09:40.097791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:40.097688 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" podUID="a1bfc658-f4c9-4280-91e9-5081b0f9416e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": read tcp 10.134.0.2:48860->10.134.0.216:15021: read: connection reset by peer" Oct 30 21:09:40.772787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:40.772646 4848 generic.go:356] "Generic (PLEG): container finished" podID="a1bfc658-f4c9-4280-91e9-5081b0f9416e" containerID="f2f691e82ac81003b1527be23474f45403729a2b5c5d1e4512e21e8d22b4f51e" exitCode=0 Oct 30 21:09:40.773023 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:40.772941 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" event={"ID":"a1bfc658-f4c9-4280-91e9-5081b0f9416e","Type":"ContainerDied","Data":"f2f691e82ac81003b1527be23474f45403729a2b5c5d1e4512e21e8d22b4f51e"} Oct 30 21:09:40.778032 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:40.773538 4848 scope.go:122] "RemoveContainer" containerID="f2f691e82ac81003b1527be23474f45403729a2b5c5d1e4512e21e8d22b4f51e" Oct 30 21:09:40.778032 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:40.776157 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:40.778032 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:40.776258 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:40.778032 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:40.776306 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:41.783857 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:41.783822 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" event={"ID":"a1bfc658-f4c9-4280-91e9-5081b0f9416e","Type":"ContainerStarted","Data":"2334048ffb6810084c8a07d391b456f6a03f946b15926efabd11aafdde563eea"} Oct 30 21:09:42.297866 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:42.297832 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:09:42.298246 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:42.298214 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 21:09:42.298328 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:42.298301 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" podUID="a1bfc658-f4c9-4280-91e9-5081b0f9416e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 21:09:43.297429 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:43.297393 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 21:09:43.297862 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:43.297446 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" podUID="a1bfc658-f4c9-4280-91e9-5081b0f9416e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 21:09:44.297956 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:44.297915 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 21:09:44.298487 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:44.297989 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" podUID="a1bfc658-f4c9-4280-91e9-5081b0f9416e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 21:09:44.848012 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:44.847966 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj"] Oct 30 21:09:44.848236 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:44.848207 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" podUID="3402ddb4-1b5f-4458-a868-19a4add51793" containerName="istio-proxy" containerID="cri-o://413f427ba2ad7bd77d8b8dfdffbf3aad21858a314cbcb70f9f2cd4352fa7648d" gracePeriod=30 Oct 30 21:09:45.304874 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:45.302149 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 21:09:45.304874 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:45.302228 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" podUID="a1bfc658-f4c9-4280-91e9-5081b0f9416e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 21:09:46.297125 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:46.297081 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 21:09:46.297343 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:46.297145 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" podUID="a1bfc658-f4c9-4280-91e9-5081b0f9416e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 21:09:47.300155 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.300119 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 21:09:47.300558 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.300179 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" podUID="a1bfc658-f4c9-4280-91e9-5081b0f9416e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 21:09:47.300671 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.300647 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq"] Oct 30 21:09:47.301868 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.301848 4848 cpu_manager.go:414] "containerMap: removing container" podUID="a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26" containerName="payload-pre-processing" Oct 30 21:09:47.301920 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.301872 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26" containerName="payload-pre-processing" Oct 30 21:09:47.302074 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.302060 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ccf619-5a46-4593-b3fc-8f3b4fd4ce26" containerName="payload-pre-processing" Oct 30 21:09:47.310635 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.310611 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.325280 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.325249 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-e8063c-openshift-default-dockercfg-kltr8\"" Oct 30 21:09:47.331491 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.331461 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq"] Oct 30 21:09:47.651887 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.645643 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-credential-socket\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.651887 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.645699 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfvvm\" (UniqueName: \"kubernetes.io/projected/11a4ae21-6152-44d6-9327-18c9a856261c-kube-api-access-wfvvm\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.651887 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.645892 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-workload-certs\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.651887 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.645939 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11a4ae21-6152-44d6-9327-18c9a856261c-istio-podinfo\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.651887 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.646034 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-istio-data\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.651887 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.646100 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-workload-socket\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.651887 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.646153 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11a4ae21-6152-44d6-9327-18c9a856261c-istiod-ca-cert\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.651887 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.646194 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-istio-envoy\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.651887 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.646232 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11a4ae21-6152-44d6-9327-18c9a856261c-istio-token\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.753696 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.749319 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-workload-certs\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.754073 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.754040 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11a4ae21-6152-44d6-9327-18c9a856261c-istio-podinfo\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.754988 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.754961 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-istio-data\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.755206 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.755188 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-workload-socket\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.755357 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.755342 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11a4ae21-6152-44d6-9327-18c9a856261c-istiod-ca-cert\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.755497 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.755481 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-istio-envoy\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.755708 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.755689 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11a4ae21-6152-44d6-9327-18c9a856261c-istio-token\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.756282 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.756259 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-credential-socket\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.756689 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.756667 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfvvm\" (UniqueName: \"kubernetes.io/projected/11a4ae21-6152-44d6-9327-18c9a856261c-kube-api-access-wfvvm\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.757043 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.756046 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-workload-socket\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.757178 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.756512 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11a4ae21-6152-44d6-9327-18c9a856261c-istiod-ca-cert\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.757284 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.757065 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-credential-socket\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.757383 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.755884 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-istio-data\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.757556 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.754324 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-workload-certs\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.769477 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.767315 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfvvm\" (UniqueName: \"kubernetes.io/projected/11a4ae21-6152-44d6-9327-18c9a856261c-kube-api-access-wfvvm\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.769669 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.768441 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11a4ae21-6152-44d6-9327-18c9a856261c-istio-token\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.769864 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.769148 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11a4ae21-6152-44d6-9327-18c9a856261c-istio-podinfo\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.770014 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.769680 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-istio-envoy\") pod \"e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:47.938282 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:47.937985 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:48.322161 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:48.321175 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:09:48.334570 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:48.334122 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:09:48.335725 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:48.335598 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:09:48.379861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:48.379836 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq"] Oct 30 21:09:48.381140 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:09:48.381117 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a4ae21_6152_44d6_9327_18c9a856261c.slice/crio-6a403d9b7fcd696c9aa920c35e341c3358b08a1d3deb9519d6e8730fb3220742 WatchSource:0}: Error finding container 6a403d9b7fcd696c9aa920c35e341c3358b08a1d3deb9519d6e8730fb3220742: Status 404 returned error can't find the container with id 6a403d9b7fcd696c9aa920c35e341c3358b08a1d3deb9519d6e8730fb3220742 Oct 30 21:09:48.384396 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:48.384361 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:48.384483 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:48.384443 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:48.384523 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:48.384482 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:48.958761 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:48.958728 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" event={"ID":"11a4ae21-6152-44d6-9327-18c9a856261c","Type":"ContainerStarted","Data":"59bbafcc5ce9a738b54f45510a81224e29f30ac77a2609f8700b6f1190dd1a82"} Oct 30 21:09:48.958959 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:48.958824 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" event={"ID":"11a4ae21-6152-44d6-9327-18c9a856261c","Type":"ContainerStarted","Data":"6a403d9b7fcd696c9aa920c35e341c3358b08a1d3deb9519d6e8730fb3220742"} Oct 30 21:09:48.985005 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:48.984958 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" podStartSLOduration=1.984943977 podStartE2EDuration="1.984943977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:48.980597985 +0000 UTC m=+1894.160494493" watchObservedRunningTime="2026-10-30 21:09:48.984943977 +0000 UTC m=+1894.164840484" Oct 30 21:09:49.662448 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:49.662414 4848 patch_prober.go:29] interesting pod/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:09:49.662872 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:49.662464 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" podUID="3402ddb4-1b5f-4458-a868-19a4add51793" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:09:49.939654 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:49.939626 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:49.945791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:49.945729 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:49.966296 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:49.966257 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:49.967220 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:49.967199 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:09:50.246623 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.246588 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:09:50.294413 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.294384 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-istio-data\") pod \"3402ddb4-1b5f-4458-a868-19a4add51793\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " Oct 30 21:09:50.294617 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.294484 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-credential-socket\") pod \"3402ddb4-1b5f-4458-a868-19a4add51793\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " Oct 30 21:09:50.294617 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.294546 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3402ddb4-1b5f-4458-a868-19a4add51793-kube-api-access-rsv86\" (UniqueName: \"kubernetes.io/projected/3402ddb4-1b5f-4458-a868-19a4add51793-kube-api-access-rsv86\") pod \"3402ddb4-1b5f-4458-a868-19a4add51793\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " Oct 30 21:09:50.294617 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.294603 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-workload-socket\") pod \"3402ddb4-1b5f-4458-a868-19a4add51793\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " Oct 30 21:09:50.294861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.294672 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3402ddb4-1b5f-4458-a868-19a4add51793-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3402ddb4-1b5f-4458-a868-19a4add51793-istiod-ca-cert\") pod \"3402ddb4-1b5f-4458-a868-19a4add51793\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " Oct 30 21:09:50.294861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.294732 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3402ddb4-1b5f-4458-a868-19a4add51793-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3402ddb4-1b5f-4458-a868-19a4add51793-istio-podinfo\") pod \"3402ddb4-1b5f-4458-a868-19a4add51793\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " Oct 30 21:09:50.294861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.294759 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-istio-data" pod "3402ddb4-1b5f-4458-a868-19a4add51793" (UID: "3402ddb4-1b5f-4458-a868-19a4add51793"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:09:50.294861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.294787 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-credential-socket" pod "3402ddb4-1b5f-4458-a868-19a4add51793" (UID: "3402ddb4-1b5f-4458-a868-19a4add51793"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:09:50.294861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.294794 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-workload-certs\") pod \"3402ddb4-1b5f-4458-a868-19a4add51793\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " Oct 30 21:09:50.295147 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.294948 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3402ddb4-1b5f-4458-a868-19a4add51793-istio-token\" (UniqueName: \"kubernetes.io/projected/3402ddb4-1b5f-4458-a868-19a4add51793-istio-token\") pod \"3402ddb4-1b5f-4458-a868-19a4add51793\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " Oct 30 21:09:50.295147 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.294956 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-workload-socket" pod "3402ddb4-1b5f-4458-a868-19a4add51793" (UID: "3402ddb4-1b5f-4458-a868-19a4add51793"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:09:50.295147 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.294985 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-workload-certs" pod "3402ddb4-1b5f-4458-a868-19a4add51793" (UID: "3402ddb4-1b5f-4458-a868-19a4add51793"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:09:50.295147 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.295014 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-istio-envoy\") pod \"3402ddb4-1b5f-4458-a868-19a4add51793\" (UID: \"3402ddb4-1b5f-4458-a868-19a4add51793\") " Oct 30 21:09:50.295147 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.295132 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3402ddb4-1b5f-4458-a868-19a4add51793-istiod-ca-cert" pod "3402ddb4-1b5f-4458-a868-19a4add51793" (UID: "3402ddb4-1b5f-4458-a868-19a4add51793"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:09:50.295843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.295816 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:09:50.295967 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.295847 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3402ddb4-1b5f-4458-a868-19a4add51793-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:09:50.295967 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.295861 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:09:50.295967 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.295878 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:09:50.295967 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.295893 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:09:50.297441 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.297407 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3402ddb4-1b5f-4458-a868-19a4add51793-istio-token" pod "3402ddb4-1b5f-4458-a868-19a4add51793" (UID: "3402ddb4-1b5f-4458-a868-19a4add51793"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:09:50.299211 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.298041 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3402ddb4-1b5f-4458-a868-19a4add51793-kube-api-access-rsv86" pod "3402ddb4-1b5f-4458-a868-19a4add51793" (UID: "3402ddb4-1b5f-4458-a868-19a4add51793"). InnerVolumeSpecName "kube-api-access-rsv86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:09:50.299211 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.299143 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3402ddb4-1b5f-4458-a868-19a4add51793-istio-podinfo" pod "3402ddb4-1b5f-4458-a868-19a4add51793" (UID: "3402ddb4-1b5f-4458-a868-19a4add51793"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:09:50.299932 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.299903 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-istio-envoy" pod "3402ddb4-1b5f-4458-a868-19a4add51793" (UID: "3402ddb4-1b5f-4458-a868-19a4add51793"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:09:50.397238 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.397201 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3402ddb4-1b5f-4458-a868-19a4add51793-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:09:50.397238 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.397231 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3402ddb4-1b5f-4458-a868-19a4add51793-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:09:50.397238 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.397241 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3402ddb4-1b5f-4458-a868-19a4add51793-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:09:50.397491 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.397249 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsv86\" (UniqueName: \"kubernetes.io/projected/3402ddb4-1b5f-4458-a868-19a4add51793-kube-api-access-rsv86\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:09:50.975008 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.974977 4848 generic.go:356] "Generic (PLEG): container finished" podID="3402ddb4-1b5f-4458-a868-19a4add51793" containerID="413f427ba2ad7bd77d8b8dfdffbf3aad21858a314cbcb70f9f2cd4352fa7648d" exitCode=0 Oct 30 21:09:50.975447 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.975022 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" Oct 30 21:09:50.975447 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.975067 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" event={"ID":"3402ddb4-1b5f-4458-a868-19a4add51793","Type":"ContainerDied","Data":"413f427ba2ad7bd77d8b8dfdffbf3aad21858a314cbcb70f9f2cd4352fa7648d"} Oct 30 21:09:50.975447 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.975122 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj" event={"ID":"3402ddb4-1b5f-4458-a868-19a4add51793","Type":"ContainerDied","Data":"0b5cc9a88d503a643135dc7d590a7135654ce4d99a88fbf7f040adc295106a73"} Oct 30 21:09:50.975447 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.975146 4848 scope.go:122] "RemoveContainer" containerID="413f427ba2ad7bd77d8b8dfdffbf3aad21858a314cbcb70f9f2cd4352fa7648d" Oct 30 21:09:50.986064 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.986037 4848 scope.go:122] "RemoveContainer" containerID="63c93aaaee2974cfa22d1f350b159ccffa635f838191e2bcb40faec6dc5e7126" Oct 30 21:09:50.997803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.996978 4848 scope.go:122] "RemoveContainer" containerID="413f427ba2ad7bd77d8b8dfdffbf3aad21858a314cbcb70f9f2cd4352fa7648d" Oct 30 21:09:50.997803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.997264 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj"] Oct 30 21:09:50.997803 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:09:50.997293 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"413f427ba2ad7bd77d8b8dfdffbf3aad21858a314cbcb70f9f2cd4352fa7648d\": container with ID starting with 413f427ba2ad7bd77d8b8dfdffbf3aad21858a314cbcb70f9f2cd4352fa7648d not found: ID does not exist" containerID="413f427ba2ad7bd77d8b8dfdffbf3aad21858a314cbcb70f9f2cd4352fa7648d" Oct 30 21:09:50.997803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.997321 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"413f427ba2ad7bd77d8b8dfdffbf3aad21858a314cbcb70f9f2cd4352fa7648d"} err="failed to get container status \"413f427ba2ad7bd77d8b8dfdffbf3aad21858a314cbcb70f9f2cd4352fa7648d\": rpc error: code = NotFound desc = could not find container \"413f427ba2ad7bd77d8b8dfdffbf3aad21858a314cbcb70f9f2cd4352fa7648d\": container with ID starting with 413f427ba2ad7bd77d8b8dfdffbf3aad21858a314cbcb70f9f2cd4352fa7648d not found: ID does not exist" Oct 30 21:09:50.997803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.997345 4848 scope.go:122] "RemoveContainer" containerID="63c93aaaee2974cfa22d1f350b159ccffa635f838191e2bcb40faec6dc5e7126" Oct 30 21:09:50.997803 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:09:50.997606 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c93aaaee2974cfa22d1f350b159ccffa635f838191e2bcb40faec6dc5e7126\": container with ID starting with 63c93aaaee2974cfa22d1f350b159ccffa635f838191e2bcb40faec6dc5e7126 not found: ID does not exist" containerID="63c93aaaee2974cfa22d1f350b159ccffa635f838191e2bcb40faec6dc5e7126" Oct 30 21:09:50.997803 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:50.997633 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c93aaaee2974cfa22d1f350b159ccffa635f838191e2bcb40faec6dc5e7126"} err="failed to get container status \"63c93aaaee2974cfa22d1f350b159ccffa635f838191e2bcb40faec6dc5e7126\": rpc error: code = NotFound desc = could not find container \"63c93aaaee2974cfa22d1f350b159ccffa635f838191e2bcb40faec6dc5e7126\": container with ID starting with 63c93aaaee2974cfa22d1f350b159ccffa635f838191e2bcb40faec6dc5e7126 not found: ID does not exist" Oct 30 21:09:51.002135 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:51.002115 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-435414-openshift-default-7b78b47f57-9lbtj"] Oct 30 21:09:52.520534 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:09:52.520489 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:09:52.984030 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:09:52.983985 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3402ddb4-1b5f-4458-a868-19a4add51793" path="/var/lib/kubelet/pods/3402ddb4-1b5f-4458-a868-19a4add51793/volumes" Oct 30 21:10:08.143951 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:08.143906 4848 patch_prober.go:29] interesting pod/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:10:08.144388 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:08.143992 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" podUID="9f01e20c-1755-4c64-9dab-f397c8fc4852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:10:10.209147 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:10.209112 4848 generic.go:356] "Generic (PLEG): container finished" podID="9f01e20c-1755-4c64-9dab-f397c8fc4852" containerID="b2fbdcbd5e87f2aed825dbbac88b44aeaca17be0f266d81918ed9c5f6219d60e" exitCode=0 Oct 30 21:10:10.209640 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:10.209173 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" event={"ID":"9f01e20c-1755-4c64-9dab-f397c8fc4852","Type":"ContainerDied","Data":"b2fbdcbd5e87f2aed825dbbac88b44aeaca17be0f266d81918ed9c5f6219d60e"} Oct 30 21:10:10.209640 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:10.209551 4848 scope.go:122] "RemoveContainer" containerID="b2fbdcbd5e87f2aed825dbbac88b44aeaca17be0f266d81918ed9c5f6219d60e" Oct 30 21:10:10.211420 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:10.211391 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:10:10.211503 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:10.211461 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:10:10.211503 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:10.211492 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:10:11.237585 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.237551 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" event={"ID":"9f01e20c-1755-4c64-9dab-f397c8fc4852","Type":"ContainerStarted","Data":"7e1ec5c2a2e1df1ae898828d5bfd808010d0c61bd1f506c88962fce98b599ff6"} Oct 30 21:10:11.285823 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.285791 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w"] Oct 30 21:10:11.290138 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.290108 4848 cpu_manager.go:414] "containerMap: removing container" podUID="3402ddb4-1b5f-4458-a868-19a4add51793" containerName="istio-proxy" Oct 30 21:10:11.290138 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.290135 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3402ddb4-1b5f-4458-a868-19a4add51793" containerName="istio-proxy" Oct 30 21:10:11.290366 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.290166 4848 cpu_manager.go:414] "containerMap: removing container" podUID="3402ddb4-1b5f-4458-a868-19a4add51793" containerName="istio-proxy" Oct 30 21:10:11.290366 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.290175 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3402ddb4-1b5f-4458-a868-19a4add51793" containerName="istio-proxy" Oct 30 21:10:11.290483 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.290411 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="3402ddb4-1b5f-4458-a868-19a4add51793" containerName="istio-proxy" Oct 30 21:10:11.290483 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.290433 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="3402ddb4-1b5f-4458-a868-19a4add51793" containerName="istio-proxy" Oct 30 21:10:11.294122 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.294062 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" Oct 30 21:10:11.300628 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.300006 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-e8063c-serving-cert\"" Oct 30 21:10:11.341475 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.341441 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w"] Oct 30 21:10:11.443649 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.442857 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5c2e61-6970-43c5-9fb4-1f3991a49466-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w\" (UID: \"ac5c2e61-6970-43c5-9fb4-1f3991a49466\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" Oct 30 21:10:11.443649 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.442920 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5c2e61-6970-43c5-9fb4-1f3991a49466-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w\" (UID: \"ac5c2e61-6970-43c5-9fb4-1f3991a49466\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" Oct 30 21:10:11.443649 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.442972 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5c2e61-6970-43c5-9fb4-1f3991a49466-metrics-certs\") pod \"maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w\" (UID: \"ac5c2e61-6970-43c5-9fb4-1f3991a49466\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" Oct 30 21:10:11.443649 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.443331 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq6pc\" (UniqueName: \"kubernetes.io/projected/ac5c2e61-6970-43c5-9fb4-1f3991a49466-kube-api-access-bq6pc\") pod \"maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w\" (UID: \"ac5c2e61-6970-43c5-9fb4-1f3991a49466\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" Oct 30 21:10:11.558889 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.554519 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5c2e61-6970-43c5-9fb4-1f3991a49466-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w\" (UID: \"ac5c2e61-6970-43c5-9fb4-1f3991a49466\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" Oct 30 21:10:11.558889 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.554571 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5c2e61-6970-43c5-9fb4-1f3991a49466-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w\" (UID: \"ac5c2e61-6970-43c5-9fb4-1f3991a49466\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" Oct 30 21:10:11.558889 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.554614 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5c2e61-6970-43c5-9fb4-1f3991a49466-metrics-certs\") pod \"maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w\" (UID: \"ac5c2e61-6970-43c5-9fb4-1f3991a49466\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" Oct 30 21:10:11.558889 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.554988 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq6pc\" (UniqueName: \"kubernetes.io/projected/ac5c2e61-6970-43c5-9fb4-1f3991a49466-kube-api-access-bq6pc\") pod \"maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w\" (UID: \"ac5c2e61-6970-43c5-9fb4-1f3991a49466\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" Oct 30 21:10:11.558889 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.556404 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5c2e61-6970-43c5-9fb4-1f3991a49466-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w\" (UID: \"ac5c2e61-6970-43c5-9fb4-1f3991a49466\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" Oct 30 21:10:11.562333 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.562306 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5c2e61-6970-43c5-9fb4-1f3991a49466-metrics-certs\") pod \"maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w\" (UID: \"ac5c2e61-6970-43c5-9fb4-1f3991a49466\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" Oct 30 21:10:11.563600 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.563571 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5c2e61-6970-43c5-9fb4-1f3991a49466-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w\" (UID: \"ac5c2e61-6970-43c5-9fb4-1f3991a49466\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" Oct 30 21:10:11.563826 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.563807 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq6pc\" (UniqueName: \"kubernetes.io/projected/ac5c2e61-6970-43c5-9fb4-1f3991a49466-kube-api-access-bq6pc\") pod \"maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w\" (UID: \"ac5c2e61-6970-43c5-9fb4-1f3991a49466\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" Oct 30 21:10:11.642052 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.642022 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" Oct 30 21:10:11.678262 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.678229 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:10:11.678465 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.678442 4848 patch_prober.go:29] interesting pod/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 30 21:10:11.678546 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:11.678500 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" podUID="9f01e20c-1755-4c64-9dab-f397c8fc4852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 30 21:10:12.068791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:12.055303 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w"] Oct 30 21:10:12.248776 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:12.248720 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" event={"ID":"ac5c2e61-6970-43c5-9fb4-1f3991a49466","Type":"ContainerStarted","Data":"f501c624e9486ddbc4273646ba3fe7b68bd0cf0b87601ecb0c75b08b9ecf3dba"} Oct 30 21:10:12.677675 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:12.677648 4848 patch_prober.go:29] interesting pod/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 30 21:10:12.677825 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:12.677698 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" podUID="9f01e20c-1755-4c64-9dab-f397c8fc4852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 30 21:10:13.261150 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:13.261109 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" event={"ID":"ac5c2e61-6970-43c5-9fb4-1f3991a49466","Type":"ContainerStarted","Data":"292d27bb2eb379c9a32140b8d71caafd0e1d42854a051676392b6f055f901584"} Oct 30 21:10:13.262125 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:13.262072 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" Oct 30 21:10:13.295276 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:13.295227 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" podStartSLOduration=1.787552148 podStartE2EDuration="2.295213722s" totalImagesPullingTime="507.661574ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:13.294018532 +0000 UTC m=+1918.473915039" watchObservedRunningTime="2026-10-30 21:10:13.295213722 +0000 UTC m=+1918.475110270" Oct 30 21:10:13.678300 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:13.678270 4848 patch_prober.go:29] interesting pod/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 30 21:10:13.679982 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:13.679945 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" podUID="9f01e20c-1755-4c64-9dab-f397c8fc4852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 30 21:10:14.678080 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:14.678044 4848 patch_prober.go:29] interesting pod/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 30 21:10:14.678586 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:14.678104 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" podUID="9f01e20c-1755-4c64-9dab-f397c8fc4852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 30 21:10:15.683583 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:15.683549 4848 patch_prober.go:29] interesting pod/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 30 21:10:15.684204 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:15.684173 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" podUID="9f01e20c-1755-4c64-9dab-f397c8fc4852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 30 21:10:16.678569 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:16.678530 4848 patch_prober.go:29] interesting pod/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body= Oct 30 21:10:16.678963 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:16.678931 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" podUID="9f01e20c-1755-4c64-9dab-f397c8fc4852" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" Oct 30 21:10:17.712349 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:17.712311 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:10:17.712821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:17.712716 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:10:17.713325 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:17.713303 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:10:20.298974 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:20.298945 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" Oct 30 21:10:22.513467 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:10:22.513415 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:10:33.546362 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:33.546329 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8"] Oct 30 21:10:33.547328 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:33.546583 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" podUID="4bb21377-14eb-4b8d-86dc-19107022c1c9" containerName="maas-api" containerID="cri-o://576124d0ab21466d91e270a8456ff89e5b32efe09cebdb386f10aec75269700f" gracePeriod=30 Oct 30 21:10:33.690847 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:33.685243 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" podUID="4bb21377-14eb-4b8d-86dc-19107022c1c9" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.220:8443/health\": dial tcp 10.134.0.220:8443: connect: connection refused" Oct 30 21:10:34.227801 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.226646 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" Oct 30 21:10:34.430905 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.429693 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bb21377-14eb-4b8d-86dc-19107022c1c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb21377-14eb-4b8d-86dc-19107022c1c9-maas-api-tls\") pod \"4bb21377-14eb-4b8d-86dc-19107022c1c9\" (UID: \"4bb21377-14eb-4b8d-86dc-19107022c1c9\") " Oct 30 21:10:34.430905 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.429856 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bb21377-14eb-4b8d-86dc-19107022c1c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb21377-14eb-4b8d-86dc-19107022c1c9-openshift-service-ca\") pod \"4bb21377-14eb-4b8d-86dc-19107022c1c9\" (UID: \"4bb21377-14eb-4b8d-86dc-19107022c1c9\") " Oct 30 21:10:34.430905 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.430078 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bb21377-14eb-4b8d-86dc-19107022c1c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb21377-14eb-4b8d-86dc-19107022c1c9-metrics-certs\") pod \"4bb21377-14eb-4b8d-86dc-19107022c1c9\" (UID: \"4bb21377-14eb-4b8d-86dc-19107022c1c9\") " Oct 30 21:10:34.430905 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.430208 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb21377-14eb-4b8d-86dc-19107022c1c9-kube-api-access-vtdrz\" (UniqueName: \"kubernetes.io/projected/4bb21377-14eb-4b8d-86dc-19107022c1c9-kube-api-access-vtdrz\") pod \"4bb21377-14eb-4b8d-86dc-19107022c1c9\" (UID: \"4bb21377-14eb-4b8d-86dc-19107022c1c9\") " Oct 30 21:10:34.431309 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.431274 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb21377-14eb-4b8d-86dc-19107022c1c9-openshift-service-ca" pod "4bb21377-14eb-4b8d-86dc-19107022c1c9" (UID: "4bb21377-14eb-4b8d-86dc-19107022c1c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:10:34.433008 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.432974 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb21377-14eb-4b8d-86dc-19107022c1c9-metrics-certs" pod "4bb21377-14eb-4b8d-86dc-19107022c1c9" (UID: "4bb21377-14eb-4b8d-86dc-19107022c1c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:10:34.433793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.433741 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb21377-14eb-4b8d-86dc-19107022c1c9-maas-api-tls" pod "4bb21377-14eb-4b8d-86dc-19107022c1c9" (UID: "4bb21377-14eb-4b8d-86dc-19107022c1c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:10:34.434080 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.434052 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb21377-14eb-4b8d-86dc-19107022c1c9-kube-api-access-vtdrz" pod "4bb21377-14eb-4b8d-86dc-19107022c1c9" (UID: "4bb21377-14eb-4b8d-86dc-19107022c1c9"). InnerVolumeSpecName "kube-api-access-vtdrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:10:34.532426 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.532384 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb21377-14eb-4b8d-86dc-19107022c1c9-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:34.532426 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.532425 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtdrz\" (UniqueName: \"kubernetes.io/projected/4bb21377-14eb-4b8d-86dc-19107022c1c9-kube-api-access-vtdrz\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:34.532662 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.532445 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb21377-14eb-4b8d-86dc-19107022c1c9-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:34.532662 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.532460 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb21377-14eb-4b8d-86dc-19107022c1c9-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:34.559025 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.558990 4848 generic.go:356] "Generic (PLEG): container finished" podID="4bb21377-14eb-4b8d-86dc-19107022c1c9" containerID="576124d0ab21466d91e270a8456ff89e5b32efe09cebdb386f10aec75269700f" exitCode=0 Oct 30 21:10:34.559541 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.559043 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" Oct 30 21:10:34.559541 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.559067 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" event={"ID":"4bb21377-14eb-4b8d-86dc-19107022c1c9","Type":"ContainerDied","Data":"576124d0ab21466d91e270a8456ff89e5b32efe09cebdb386f10aec75269700f"} Oct 30 21:10:34.559541 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.559134 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8" event={"ID":"4bb21377-14eb-4b8d-86dc-19107022c1c9","Type":"ContainerDied","Data":"9697350553c33aa64d3b41b3960937e8570a799f1f8eb9cec9fdc815dc10297e"} Oct 30 21:10:34.559541 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.559158 4848 scope.go:122] "RemoveContainer" containerID="576124d0ab21466d91e270a8456ff89e5b32efe09cebdb386f10aec75269700f" Oct 30 21:10:34.588800 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.588781 4848 scope.go:122] "RemoveContainer" containerID="576124d0ab21466d91e270a8456ff89e5b32efe09cebdb386f10aec75269700f" Oct 30 21:10:34.589131 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:10:34.589112 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"576124d0ab21466d91e270a8456ff89e5b32efe09cebdb386f10aec75269700f\": container with ID starting with 576124d0ab21466d91e270a8456ff89e5b32efe09cebdb386f10aec75269700f not found: ID does not exist" containerID="576124d0ab21466d91e270a8456ff89e5b32efe09cebdb386f10aec75269700f" Oct 30 21:10:34.589192 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.589143 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"576124d0ab21466d91e270a8456ff89e5b32efe09cebdb386f10aec75269700f"} err="failed to get container status \"576124d0ab21466d91e270a8456ff89e5b32efe09cebdb386f10aec75269700f\": rpc error: code = NotFound desc = could not find container \"576124d0ab21466d91e270a8456ff89e5b32efe09cebdb386f10aec75269700f\": container with ID starting with 576124d0ab21466d91e270a8456ff89e5b32efe09cebdb386f10aec75269700f not found: ID does not exist" Oct 30 21:10:34.602399 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.602373 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8"] Oct 30 21:10:34.630830 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.629414 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-7386b4-794874bbbf-hwmw8"] Oct 30 21:10:34.934732 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:34.934700 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb21377-14eb-4b8d-86dc-19107022c1c9" path="/var/lib/kubelet/pods/4bb21377-14eb-4b8d-86dc-19107022c1c9/volumes" Oct 30 21:10:36.007895 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:36.007856 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:10:36.008326 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:36.007919 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" podUID="11a4ae21-6152-44d6-9327-18c9a856261c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:10:40.617518 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:40.617482 4848 generic.go:356] "Generic (PLEG): container finished" podID="11a4ae21-6152-44d6-9327-18c9a856261c" containerID="59bbafcc5ce9a738b54f45510a81224e29f30ac77a2609f8700b6f1190dd1a82" exitCode=0 Oct 30 21:10:40.617996 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:40.617551 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" event={"ID":"11a4ae21-6152-44d6-9327-18c9a856261c","Type":"ContainerDied","Data":"59bbafcc5ce9a738b54f45510a81224e29f30ac77a2609f8700b6f1190dd1a82"} Oct 30 21:10:40.618054 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:40.618037 4848 scope.go:122] "RemoveContainer" containerID="59bbafcc5ce9a738b54f45510a81224e29f30ac77a2609f8700b6f1190dd1a82" Oct 30 21:10:40.620145 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:40.620111 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:10:40.620265 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:40.620194 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:10:40.620265 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:40.620231 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:10:41.640591 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:41.640552 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" event={"ID":"11a4ae21-6152-44d6-9327-18c9a856261c","Type":"ContainerStarted","Data":"1588523f2f7b16543e4b924f7b5b5e26b2f8420a4a5d6e4aa331c71882155d73"} Oct 30 21:10:41.940138 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:41.940102 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:10:41.940493 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:41.940432 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 21:10:41.940658 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:41.940504 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" podUID="11a4ae21-6152-44d6-9327-18c9a856261c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 21:10:42.237039 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:42.236944 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k"] Oct 30 21:10:42.237241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:42.237212 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" podUID="9f01e20c-1755-4c64-9dab-f397c8fc4852" containerName="istio-proxy" containerID="cri-o://7e1ec5c2a2e1df1ae898828d5bfd808010d0c61bd1f506c88962fce98b599ff6" gracePeriod=30 Oct 30 21:10:42.939792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:42.939731 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 21:10:42.940382 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:42.940349 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" podUID="11a4ae21-6152-44d6-9327-18c9a856261c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 21:10:44.076821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.032522 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 21:10:44.077917 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.077877 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" podUID="11a4ae21-6152-44d6-9327-18c9a856261c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 21:10:44.437830 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.437787 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx"] Oct 30 21:10:44.438690 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.438661 4848 cpu_manager.go:414] "containerMap: removing container" podUID="4bb21377-14eb-4b8d-86dc-19107022c1c9" containerName="maas-api" Oct 30 21:10:44.438834 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.438819 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb21377-14eb-4b8d-86dc-19107022c1c9" containerName="maas-api" Oct 30 21:10:44.439026 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.439010 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb21377-14eb-4b8d-86dc-19107022c1c9" containerName="maas-api" Oct 30 21:10:44.442301 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.442280 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.452923 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.451051 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-00909f-openshift-default-dockercfg-t87br\"" Oct 30 21:10:44.459964 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.459937 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx"] Oct 30 21:10:44.643787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.634109 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksprd\" (UniqueName: \"kubernetes.io/projected/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-kube-api-access-ksprd\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.643787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.634344 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-token\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.643787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.634481 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-podinfo\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.643787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.634532 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istiod-ca-cert\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.643787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.634623 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-workload-certs\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.643787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.634806 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-data\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.643787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.634890 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-envoy\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.643787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.634975 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-credential-socket\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.643787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.635044 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-workload-socket\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.746861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.741401 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksprd\" (UniqueName: \"kubernetes.io/projected/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-kube-api-access-ksprd\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.746861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.741562 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-token\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.746861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.741658 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-podinfo\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.746861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.741707 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istiod-ca-cert\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.746861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.741759 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-workload-certs\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.746861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.741870 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-data\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.746861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.741947 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-envoy\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.746861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.742009 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-credential-socket\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.746861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.742060 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-workload-socket\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.746861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.742684 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-workload-socket\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.746861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.744450 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-credential-socket\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.746861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.744806 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-workload-certs\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.746861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.745413 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istiod-ca-cert\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.765858 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.763374 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-data\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.797794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.797626 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-token\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.798406 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.798354 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-podinfo\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.799073 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.799042 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-envoy\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.801105 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.801082 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksprd\" (UniqueName: \"kubernetes.io/projected/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-kube-api-access-ksprd\") pod \"e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:44.947109 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.947070 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 21:10:44.948205 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:44.948172 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" podUID="11a4ae21-6152-44d6-9327-18c9a856261c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 21:10:45.077324 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:45.077242 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:45.442269 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:45.442223 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx"] Oct 30 21:10:45.442555 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:10:45.442530 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cfb5025_0b7e_4697_bd4e_e65fff80e66c.slice/crio-8da43e92c868dcda8477f9f7dd721513d11a4110430346db6e27c7a265956b52 WatchSource:0}: Error finding container 8da43e92c868dcda8477f9f7dd721513d11a4110430346db6e27c7a265956b52: Status 404 returned error can't find the container with id 8da43e92c868dcda8477f9f7dd721513d11a4110430346db6e27c7a265956b52 Oct 30 21:10:45.446241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:45.446205 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:10:45.446358 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:45.446286 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:10:45.446358 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:45.446327 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:10:45.751326 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:45.751215 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" event={"ID":"4cfb5025-0b7e-4697-bd4e-e65fff80e66c","Type":"ContainerStarted","Data":"a953cf9b73616a9d21a449287deb844632e45d3f9766946e80557897406ade79"} Oct 30 21:10:45.751513 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:45.751398 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" event={"ID":"4cfb5025-0b7e-4697-bd4e-e65fff80e66c","Type":"ContainerStarted","Data":"8da43e92c868dcda8477f9f7dd721513d11a4110430346db6e27c7a265956b52"} Oct 30 21:10:45.788690 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:45.788633 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" podStartSLOduration=1.78860925 podStartE2EDuration="1.78860925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:45.785836839 +0000 UTC m=+1950.965733348" watchObservedRunningTime="2026-10-30 21:10:45.78860925 +0000 UTC m=+1950.968505756" Oct 30 21:10:45.945808 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:45.940178 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 21:10:45.945808 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:45.940250 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" podUID="11a4ae21-6152-44d6-9327-18c9a856261c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 21:10:46.077612 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:46.077532 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:46.079531 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:46.079499 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 21:10:46.079666 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:46.079562 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" podUID="4cfb5025-0b7e-4697-bd4e-e65fff80e66c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 21:10:46.940695 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:46.940353 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 21:10:46.940695 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:46.940408 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" podUID="11a4ae21-6152-44d6-9327-18c9a856261c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 21:10:47.083591 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.083549 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:47.542938 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.542905 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:10:47.712871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.712158 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-workload-socket" pod "9f01e20c-1755-4c64-9dab-f397c8fc4852" (UID: "9f01e20c-1755-4c64-9dab-f397c8fc4852"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:10:47.712871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.712230 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-workload-socket\") pod \"9f01e20c-1755-4c64-9dab-f397c8fc4852\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " Oct 30 21:10:47.712871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.712376 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-token\" (UniqueName: \"kubernetes.io/projected/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-token\") pod \"9f01e20c-1755-4c64-9dab-f397c8fc4852\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " Oct 30 21:10:47.712871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.712447 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-workload-certs\") pod \"9f01e20c-1755-4c64-9dab-f397c8fc4852\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " Oct 30 21:10:47.712871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.712570 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-envoy\") pod \"9f01e20c-1755-4c64-9dab-f397c8fc4852\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " Oct 30 21:10:47.712871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.712608 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f01e20c-1755-4c64-9dab-f397c8fc4852-kube-api-access-wwp99\" (UniqueName: \"kubernetes.io/projected/9f01e20c-1755-4c64-9dab-f397c8fc4852-kube-api-access-wwp99\") pod \"9f01e20c-1755-4c64-9dab-f397c8fc4852\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " Oct 30 21:10:47.712871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.712681 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-podinfo\") pod \"9f01e20c-1755-4c64-9dab-f397c8fc4852\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " Oct 30 21:10:47.712871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.712715 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-data\") pod \"9f01e20c-1755-4c64-9dab-f397c8fc4852\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " Oct 30 21:10:47.712871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.712791 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f01e20c-1755-4c64-9dab-f397c8fc4852-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f01e20c-1755-4c64-9dab-f397c8fc4852-istiod-ca-cert\") pod \"9f01e20c-1755-4c64-9dab-f397c8fc4852\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " Oct 30 21:10:47.712871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.712824 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-credential-socket\") pod \"9f01e20c-1755-4c64-9dab-f397c8fc4852\" (UID: \"9f01e20c-1755-4c64-9dab-f397c8fc4852\") " Oct 30 21:10:47.726492 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.726447 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-credential-socket" pod "9f01e20c-1755-4c64-9dab-f397c8fc4852" (UID: "9f01e20c-1755-4c64-9dab-f397c8fc4852"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:10:47.733967 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.733893 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f01e20c-1755-4c64-9dab-f397c8fc4852-istiod-ca-cert" pod "9f01e20c-1755-4c64-9dab-f397c8fc4852" (UID: "9f01e20c-1755-4c64-9dab-f397c8fc4852"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:10:47.734838 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.734747 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-workload-certs" pod "9f01e20c-1755-4c64-9dab-f397c8fc4852" (UID: "9f01e20c-1755-4c64-9dab-f397c8fc4852"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:10:47.735114 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.735075 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-data" pod "9f01e20c-1755-4c64-9dab-f397c8fc4852" (UID: "9f01e20c-1755-4c64-9dab-f397c8fc4852"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:10:47.738094 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.738063 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-podinfo" pod "9f01e20c-1755-4c64-9dab-f397c8fc4852" (UID: "9f01e20c-1755-4c64-9dab-f397c8fc4852"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:10:47.740241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.738443 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-envoy" pod "9f01e20c-1755-4c64-9dab-f397c8fc4852" (UID: "9f01e20c-1755-4c64-9dab-f397c8fc4852"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:10:47.741663 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.741610 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f01e20c-1755-4c64-9dab-f397c8fc4852-kube-api-access-wwp99" pod "9f01e20c-1755-4c64-9dab-f397c8fc4852" (UID: "9f01e20c-1755-4c64-9dab-f397c8fc4852"). InnerVolumeSpecName "kube-api-access-wwp99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:10:47.742712 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.742690 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-token" pod "9f01e20c-1755-4c64-9dab-f397c8fc4852" (UID: "9f01e20c-1755-4c64-9dab-f397c8fc4852"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:10:47.770732 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.770646 4848 generic.go:356] "Generic (PLEG): container finished" podID="9f01e20c-1755-4c64-9dab-f397c8fc4852" containerID="7e1ec5c2a2e1df1ae898828d5bfd808010d0c61bd1f506c88962fce98b599ff6" exitCode=0 Oct 30 21:10:47.770732 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.770711 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" Oct 30 21:10:47.770942 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.770708 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" event={"ID":"9f01e20c-1755-4c64-9dab-f397c8fc4852","Type":"ContainerDied","Data":"7e1ec5c2a2e1df1ae898828d5bfd808010d0c61bd1f506c88962fce98b599ff6"} Oct 30 21:10:47.770942 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.770887 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k" event={"ID":"9f01e20c-1755-4c64-9dab-f397c8fc4852","Type":"ContainerDied","Data":"7e1d2a626dd2e8a2c4fb203bb5d998f83df28766f400ae6be7a03348a490e855"} Oct 30 21:10:47.771049 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.770952 4848 scope.go:122] "RemoveContainer" containerID="7e1ec5c2a2e1df1ae898828d5bfd808010d0c61bd1f506c88962fce98b599ff6" Oct 30 21:10:47.771471 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.771363 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:47.772590 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.772573 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:10:47.783841 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.783824 4848 scope.go:122] "RemoveContainer" containerID="b2fbdcbd5e87f2aed825dbbac88b44aeaca17be0f266d81918ed9c5f6219d60e" Oct 30 21:10:47.793909 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.793892 4848 scope.go:122] "RemoveContainer" containerID="7e1ec5c2a2e1df1ae898828d5bfd808010d0c61bd1f506c88962fce98b599ff6" Oct 30 21:10:47.794260 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:10:47.794239 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1ec5c2a2e1df1ae898828d5bfd808010d0c61bd1f506c88962fce98b599ff6\": container with ID starting with 7e1ec5c2a2e1df1ae898828d5bfd808010d0c61bd1f506c88962fce98b599ff6 not found: ID does not exist" containerID="7e1ec5c2a2e1df1ae898828d5bfd808010d0c61bd1f506c88962fce98b599ff6" Oct 30 21:10:47.794347 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.794272 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1ec5c2a2e1df1ae898828d5bfd808010d0c61bd1f506c88962fce98b599ff6"} err="failed to get container status \"7e1ec5c2a2e1df1ae898828d5bfd808010d0c61bd1f506c88962fce98b599ff6\": rpc error: code = NotFound desc = could not find container \"7e1ec5c2a2e1df1ae898828d5bfd808010d0c61bd1f506c88962fce98b599ff6\": container with ID starting with 7e1ec5c2a2e1df1ae898828d5bfd808010d0c61bd1f506c88962fce98b599ff6 not found: ID does not exist" Oct 30 21:10:47.794347 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.794294 4848 scope.go:122] "RemoveContainer" containerID="b2fbdcbd5e87f2aed825dbbac88b44aeaca17be0f266d81918ed9c5f6219d60e" Oct 30 21:10:47.794683 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:10:47.794653 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2fbdcbd5e87f2aed825dbbac88b44aeaca17be0f266d81918ed9c5f6219d60e\": container with ID starting with b2fbdcbd5e87f2aed825dbbac88b44aeaca17be0f266d81918ed9c5f6219d60e not found: ID does not exist" containerID="b2fbdcbd5e87f2aed825dbbac88b44aeaca17be0f266d81918ed9c5f6219d60e" Oct 30 21:10:47.794761 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.794691 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2fbdcbd5e87f2aed825dbbac88b44aeaca17be0f266d81918ed9c5f6219d60e"} err="failed to get container status \"b2fbdcbd5e87f2aed825dbbac88b44aeaca17be0f266d81918ed9c5f6219d60e\": rpc error: code = NotFound desc = could not find container \"b2fbdcbd5e87f2aed825dbbac88b44aeaca17be0f266d81918ed9c5f6219d60e\": container with ID starting with b2fbdcbd5e87f2aed825dbbac88b44aeaca17be0f266d81918ed9c5f6219d60e not found: ID does not exist" Oct 30 21:10:47.823938 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.823875 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k"] Oct 30 21:10:47.825995 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.825965 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:47.826114 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.825998 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f01e20c-1755-4c64-9dab-f397c8fc4852-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:47.826114 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.826015 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:47.826114 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.826029 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:47.826114 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.826044 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:47.826114 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.826058 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:47.826114 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.826071 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:47.826114 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.826084 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwp99\" (UniqueName: \"kubernetes.io/projected/9f01e20c-1755-4c64-9dab-f397c8fc4852-kube-api-access-wwp99\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:47.826114 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.826099 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f01e20c-1755-4c64-9dab-f397c8fc4852-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:10:47.829944 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.829911 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-7386b4-openshift-default-86696945cc-hbp6k"] Oct 30 21:10:47.946918 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.946886 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:10:47.947409 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.947386 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:10:47.948369 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:47.948346 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:10:48.934405 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:10:48.934373 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f01e20c-1755-4c64-9dab-f397c8fc4852" path="/var/lib/kubelet/pods/9f01e20c-1755-4c64-9dab-f397c8fc4852/volumes" Oct 30 21:10:52.512218 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:10:52.512184 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:11:01.552271 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.542436 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68"] Oct 30 21:11:01.553959 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.553928 4848 cpu_manager.go:414] "containerMap: removing container" podUID="9f01e20c-1755-4c64-9dab-f397c8fc4852" containerName="istio-proxy" Oct 30 21:11:01.554133 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.554118 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f01e20c-1755-4c64-9dab-f397c8fc4852" containerName="istio-proxy" Oct 30 21:11:01.554490 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.554471 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f01e20c-1755-4c64-9dab-f397c8fc4852" containerName="istio-proxy" Oct 30 21:11:01.554616 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.554604 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f01e20c-1755-4c64-9dab-f397c8fc4852" containerName="istio-proxy" Oct 30 21:11:01.558134 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.558107 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68"] Oct 30 21:11:01.558337 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.558148 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" Oct 30 21:11:01.563054 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.563035 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-e1e0f5-serving-cert\"" Oct 30 21:11:01.639645 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.639598 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/248aa1ff-dd7f-455d-a0db-8881a9af59da-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68\" (UID: \"248aa1ff-dd7f-455d-a0db-8881a9af59da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" Oct 30 21:11:01.639884 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.639752 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkl7l\" (UniqueName: \"kubernetes.io/projected/248aa1ff-dd7f-455d-a0db-8881a9af59da-kube-api-access-tkl7l\") pod \"maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68\" (UID: \"248aa1ff-dd7f-455d-a0db-8881a9af59da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" Oct 30 21:11:01.639884 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.639827 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/248aa1ff-dd7f-455d-a0db-8881a9af59da-metrics-certs\") pod \"maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68\" (UID: \"248aa1ff-dd7f-455d-a0db-8881a9af59da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" Oct 30 21:11:01.640168 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.640143 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/248aa1ff-dd7f-455d-a0db-8881a9af59da-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68\" (UID: \"248aa1ff-dd7f-455d-a0db-8881a9af59da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" Oct 30 21:11:01.753794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.753289 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/248aa1ff-dd7f-455d-a0db-8881a9af59da-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68\" (UID: \"248aa1ff-dd7f-455d-a0db-8881a9af59da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" Oct 30 21:11:01.753794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.753430 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/248aa1ff-dd7f-455d-a0db-8881a9af59da-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68\" (UID: \"248aa1ff-dd7f-455d-a0db-8881a9af59da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" Oct 30 21:11:01.753794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.753517 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkl7l\" (UniqueName: \"kubernetes.io/projected/248aa1ff-dd7f-455d-a0db-8881a9af59da-kube-api-access-tkl7l\") pod \"maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68\" (UID: \"248aa1ff-dd7f-455d-a0db-8881a9af59da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" Oct 30 21:11:01.753794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.753569 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/248aa1ff-dd7f-455d-a0db-8881a9af59da-metrics-certs\") pod \"maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68\" (UID: \"248aa1ff-dd7f-455d-a0db-8881a9af59da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" Oct 30 21:11:01.759795 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.755409 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/248aa1ff-dd7f-455d-a0db-8881a9af59da-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68\" (UID: \"248aa1ff-dd7f-455d-a0db-8881a9af59da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" Oct 30 21:11:01.787790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.787302 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkl7l\" (UniqueName: \"kubernetes.io/projected/248aa1ff-dd7f-455d-a0db-8881a9af59da-kube-api-access-tkl7l\") pod \"maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68\" (UID: \"248aa1ff-dd7f-455d-a0db-8881a9af59da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" Oct 30 21:11:01.791824 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.790339 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/248aa1ff-dd7f-455d-a0db-8881a9af59da-metrics-certs\") pod \"maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68\" (UID: \"248aa1ff-dd7f-455d-a0db-8881a9af59da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" Oct 30 21:11:01.809847 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.803582 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/248aa1ff-dd7f-455d-a0db-8881a9af59da-maas-api-tls\") pod \"maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68\" (UID: \"248aa1ff-dd7f-455d-a0db-8881a9af59da\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" Oct 30 21:11:01.887199 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:01.887163 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" Oct 30 21:11:02.191563 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:11:02.191528 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248aa1ff_dd7f_455d_a0db_8881a9af59da.slice/crio-53019e5f6ffb1e60bcc35c93d0c0afdfdd07142ddb8e109c488e3cf1008f576d WatchSource:0}: Error finding container 53019e5f6ffb1e60bcc35c93d0c0afdfdd07142ddb8e109c488e3cf1008f576d: Status 404 returned error can't find the container with id 53019e5f6ffb1e60bcc35c93d0c0afdfdd07142ddb8e109c488e3cf1008f576d Oct 30 21:11:02.192210 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:02.192111 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68"] Oct 30 21:11:02.948918 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:02.948864 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" event={"ID":"248aa1ff-dd7f-455d-a0db-8881a9af59da","Type":"ContainerStarted","Data":"4701e452041447b409e68b7cf5f2a40594bdc93f0fa83d3fe3bfe70ab05a0261"} Oct 30 21:11:02.949354 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:02.948966 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" event={"ID":"248aa1ff-dd7f-455d-a0db-8881a9af59da","Type":"ContainerStarted","Data":"53019e5f6ffb1e60bcc35c93d0c0afdfdd07142ddb8e109c488e3cf1008f576d"} Oct 30 21:11:02.949354 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:02.948996 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" Oct 30 21:11:02.968993 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:02.968938 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" podStartSLOduration=1.412711214 podStartE2EDuration="1.968923267s" totalImagesPullingTime="556.212053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:02.96807879 +0000 UTC m=+1968.147975297" watchObservedRunningTime="2026-10-30 21:11:02.968923267 +0000 UTC m=+1968.148819848" Oct 30 21:11:08.976084 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:08.976049 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" Oct 30 21:11:22.512666 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:11:22.512627 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:11:30.345894 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.341691 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz"] Oct 30 21:11:30.345894 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.342929 4848 cpu_manager.go:414] "containerMap: removing container" podUID="9f01e20c-1755-4c64-9dab-f397c8fc4852" containerName="istio-proxy" Oct 30 21:11:30.345894 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.342951 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f01e20c-1755-4c64-9dab-f397c8fc4852" containerName="istio-proxy" Oct 30 21:11:30.347711 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.347684 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.353981 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.353950 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz"] Oct 30 21:11:30.356785 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.356745 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-f147de-openshift-default-dockercfg-6k89x\"" Oct 30 21:11:30.485283 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.484032 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-envoy\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.486861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.486831 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-podinfo\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.487237 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.487214 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-workload-socket\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.487520 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.487503 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-token\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.487688 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.487672 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-credential-socket\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.487889 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.487872 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-workload-certs\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.488215 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.488197 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djlz6\" (UniqueName: \"kubernetes.io/projected/29d99320-f1a9-4178-97f6-dc0a088aec0b-kube-api-access-djlz6\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.488396 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.488380 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29d99320-f1a9-4178-97f6-dc0a088aec0b-istiod-ca-cert\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.488733 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.488712 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-data\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.590892 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.590854 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djlz6\" (UniqueName: \"kubernetes.io/projected/29d99320-f1a9-4178-97f6-dc0a088aec0b-kube-api-access-djlz6\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.591166 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.591145 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29d99320-f1a9-4178-97f6-dc0a088aec0b-istiod-ca-cert\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.591315 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.591297 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-data\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.591507 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.591490 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-envoy\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.591653 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.591638 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-podinfo\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.591793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.591761 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-workload-socket\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.592024 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.592008 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-token\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.592144 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.592129 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-credential-socket\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.592287 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.592273 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-workload-certs\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.603333 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.603257 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-envoy\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.609793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.604657 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29d99320-f1a9-4178-97f6-dc0a088aec0b-istiod-ca-cert\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.617925 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.614262 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-data\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.618504 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.618472 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-workload-socket\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.634801 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.619363 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-credential-socket\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.634801 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.621156 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-workload-certs\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.638884 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.637661 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djlz6\" (UniqueName: \"kubernetes.io/projected/29d99320-f1a9-4178-97f6-dc0a088aec0b-kube-api-access-djlz6\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.664790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.648820 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-token\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.664790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.657462 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-podinfo\") pod \"e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:30.690813 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:30.690484 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:31.250128 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:31.249694 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz"] Oct 30 21:11:31.270295 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:31.270247 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:11:31.270473 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:31.270349 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:11:31.270473 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:31.270395 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:11:31.350282 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:31.350246 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" event={"ID":"29d99320-f1a9-4178-97f6-dc0a088aec0b","Type":"ContainerStarted","Data":"b5b2c2f72705d16b890685f7993a209a5b848fd055f8bdb25bf1436ba1b80687"} Oct 30 21:11:32.361418 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:32.361372 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" event={"ID":"29d99320-f1a9-4178-97f6-dc0a088aec0b","Type":"ContainerStarted","Data":"599c164945c503ee6b12beb029a0e13cfc01e722e92460d0a2d32d4c42e01dab"} Oct 30 21:11:32.388280 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:32.388214 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" podStartSLOduration=2.388193181 podStartE2EDuration="2.388193181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:32.384227922 +0000 UTC m=+1997.564124430" watchObservedRunningTime="2026-10-30 21:11:32.388193181 +0000 UTC m=+1997.568089688" Oct 30 21:11:32.691225 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:32.691185 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:32.698303 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:32.698272 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:33.369474 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:33.369443 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:33.370706 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:33.370684 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:11:41.498398 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:41.498366 4848 generic.go:356] "Generic (PLEG): container finished" podID="4cfb5025-0b7e-4697-bd4e-e65fff80e66c" containerID="a953cf9b73616a9d21a449287deb844632e45d3f9766946e80557897406ade79" exitCode=0 Oct 30 21:11:41.498871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:41.498441 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" event={"ID":"4cfb5025-0b7e-4697-bd4e-e65fff80e66c","Type":"ContainerDied","Data":"a953cf9b73616a9d21a449287deb844632e45d3f9766946e80557897406ade79"} Oct 30 21:11:41.498954 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:41.498938 4848 scope.go:122] "RemoveContainer" containerID="a953cf9b73616a9d21a449287deb844632e45d3f9766946e80557897406ade79" Oct 30 21:11:41.500809 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:41.500784 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:11:41.500876 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:41.500856 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:11:41.500914 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:41.500899 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:11:42.511872 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:42.510597 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" event={"ID":"4cfb5025-0b7e-4697-bd4e-e65fff80e66c","Type":"ContainerStarted","Data":"3932125bafc95caf06c283283e6de914f9eadc6ad43785be1145c4931decb0de"} Oct 30 21:11:43.077408 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:43.077369 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:11:43.078559 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:43.078535 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 21:11:43.078676 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:43.078598 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" podUID="4cfb5025-0b7e-4697-bd4e-e65fff80e66c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 21:11:44.077933 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:44.077735 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 21:11:44.077933 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:44.077807 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" podUID="4cfb5025-0b7e-4697-bd4e-e65fff80e66c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 21:11:45.078160 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:45.078124 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 21:11:45.078635 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:45.078610 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" podUID="4cfb5025-0b7e-4697-bd4e-e65fff80e66c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 21:11:46.093791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:46.085215 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 21:11:46.093791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:46.085283 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" podUID="4cfb5025-0b7e-4697-bd4e-e65fff80e66c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 21:11:47.077578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:47.077547 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 21:11:47.077797 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:47.077618 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" podUID="4cfb5025-0b7e-4697-bd4e-e65fff80e66c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 21:11:47.777909 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:47.777874 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:11:48.078138 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:48.078048 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Oct 30 21:11:48.078138 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:48.078102 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" podUID="4cfb5025-0b7e-4697-bd4e-e65fff80e66c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Oct 30 21:11:49.083110 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:49.083074 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:11:49.084395 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:49.084363 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:11:52.551280 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:11:52.551243 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:11:53.777737 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:53.777699 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl"] Oct 30 21:11:53.782574 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:53.782547 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" Oct 30 21:11:53.802611 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:53.802577 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-f147de-serving-cert\"" Oct 30 21:11:53.808402 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:53.808372 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl"] Oct 30 21:11:53.872448 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:53.872416 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81767769-741f-4780-93d2-32406b2a6ba6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl\" (UID: \"81767769-741f-4780-93d2-32406b2a6ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" Oct 30 21:11:53.872662 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:53.872615 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81767769-741f-4780-93d2-32406b2a6ba6-metrics-certs\") pod \"maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl\" (UID: \"81767769-741f-4780-93d2-32406b2a6ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" Oct 30 21:11:53.872662 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:53.872655 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81767769-741f-4780-93d2-32406b2a6ba6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl\" (UID: \"81767769-741f-4780-93d2-32406b2a6ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" Oct 30 21:11:53.872794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:53.872728 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxxtj\" (UniqueName: \"kubernetes.io/projected/81767769-741f-4780-93d2-32406b2a6ba6-kube-api-access-zxxtj\") pod \"maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl\" (UID: \"81767769-741f-4780-93d2-32406b2a6ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" Oct 30 21:11:53.978791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:53.976586 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81767769-741f-4780-93d2-32406b2a6ba6-metrics-certs\") pod \"maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl\" (UID: \"81767769-741f-4780-93d2-32406b2a6ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" Oct 30 21:11:53.978791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:53.976675 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81767769-741f-4780-93d2-32406b2a6ba6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl\" (UID: \"81767769-741f-4780-93d2-32406b2a6ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" Oct 30 21:11:53.978791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:53.976729 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxxtj\" (UniqueName: \"kubernetes.io/projected/81767769-741f-4780-93d2-32406b2a6ba6-kube-api-access-zxxtj\") pod \"maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl\" (UID: \"81767769-741f-4780-93d2-32406b2a6ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" Oct 30 21:11:53.978791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:53.976972 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81767769-741f-4780-93d2-32406b2a6ba6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl\" (UID: \"81767769-741f-4780-93d2-32406b2a6ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" Oct 30 21:11:53.994171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:53.991651 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81767769-741f-4780-93d2-32406b2a6ba6-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl\" (UID: \"81767769-741f-4780-93d2-32406b2a6ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" Oct 30 21:11:54.018801 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:54.015887 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81767769-741f-4780-93d2-32406b2a6ba6-metrics-certs\") pod \"maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl\" (UID: \"81767769-741f-4780-93d2-32406b2a6ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" Oct 30 21:11:54.019589 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:54.019557 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81767769-741f-4780-93d2-32406b2a6ba6-maas-api-tls\") pod \"maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl\" (UID: \"81767769-741f-4780-93d2-32406b2a6ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" Oct 30 21:11:54.039212 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:54.037319 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxxtj\" (UniqueName: \"kubernetes.io/projected/81767769-741f-4780-93d2-32406b2a6ba6-kube-api-access-zxxtj\") pod \"maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl\" (UID: \"81767769-741f-4780-93d2-32406b2a6ba6\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" Oct 30 21:11:54.117485 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:54.117454 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" Oct 30 21:11:54.400146 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:54.400112 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl"] Oct 30 21:11:54.401903 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:11:54.401880 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81767769_741f_4780_93d2_32406b2a6ba6.slice/crio-2de7698b30eef08521ae51ee9dd71e2ca1dff90e028f63121b55621b96551c2d WatchSource:0}: Error finding container 2de7698b30eef08521ae51ee9dd71e2ca1dff90e028f63121b55621b96551c2d: Status 404 returned error can't find the container with id 2de7698b30eef08521ae51ee9dd71e2ca1dff90e028f63121b55621b96551c2d Oct 30 21:11:54.700624 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:54.700577 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" event={"ID":"81767769-741f-4780-93d2-32406b2a6ba6","Type":"ContainerStarted","Data":"2de7698b30eef08521ae51ee9dd71e2ca1dff90e028f63121b55621b96551c2d"} Oct 30 21:11:55.757826 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:55.757693 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" event={"ID":"81767769-741f-4780-93d2-32406b2a6ba6","Type":"ContainerStarted","Data":"2a1b556246198b87bb5b6648178f296219ca8c45780b3669a51cc2d04d22628c"} Oct 30 21:11:55.758905 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:55.758882 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" Oct 30 21:11:55.779232 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:11:55.779187 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" podStartSLOduration=2.346319782 podStartE2EDuration="2.779173486s" totalImagesPullingTime="432.853704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:55.777578739 +0000 UTC m=+2020.957475246" watchObservedRunningTime="2026-10-30 21:11:55.779173486 +0000 UTC m=+2020.959069993" Oct 30 21:12:01.771098 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:01.771064 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" Oct 30 21:12:22.514759 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:12:22.514714 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:12:26.197643 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:26.197611 4848 generic.go:356] "Generic (PLEG): container finished" podID="29d99320-f1a9-4178-97f6-dc0a088aec0b" containerID="599c164945c503ee6b12beb029a0e13cfc01e722e92460d0a2d32d4c42e01dab" exitCode=0 Oct 30 21:12:26.198118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:26.197693 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" event={"ID":"29d99320-f1a9-4178-97f6-dc0a088aec0b","Type":"ContainerDied","Data":"599c164945c503ee6b12beb029a0e13cfc01e722e92460d0a2d32d4c42e01dab"} Oct 30 21:12:26.198177 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:26.198138 4848 scope.go:122] "RemoveContainer" containerID="599c164945c503ee6b12beb029a0e13cfc01e722e92460d0a2d32d4c42e01dab" Oct 30 21:12:26.199867 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:26.199834 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:12:26.199956 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:26.199902 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:12:26.199956 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:26.199940 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:12:27.212753 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:27.212707 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" event={"ID":"29d99320-f1a9-4178-97f6-dc0a088aec0b","Type":"ContainerStarted","Data":"6d17cc35eab33959c958b60194c8ccfafc9e267fb5df3c96d68a0aa503c6f2ab"} Oct 30 21:12:27.699788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:27.691431 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:12:27.699788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:27.691579 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 21:12:27.699788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:27.691622 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" podUID="29d99320-f1a9-4178-97f6-dc0a088aec0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 21:12:28.696789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:28.694250 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 21:12:28.696789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:28.694327 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" podUID="29d99320-f1a9-4178-97f6-dc0a088aec0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 21:12:29.691687 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:29.691649 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 21:12:29.692001 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:29.691972 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" podUID="29d99320-f1a9-4178-97f6-dc0a088aec0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 21:12:30.691881 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:30.691852 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 21:12:30.692291 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:30.691903 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" podUID="29d99320-f1a9-4178-97f6-dc0a088aec0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 21:12:31.707216 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:31.707165 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 21:12:31.707860 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:31.707819 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" podUID="29d99320-f1a9-4178-97f6-dc0a088aec0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 21:12:32.692315 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:32.692275 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 21:12:32.692519 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:32.692345 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" podUID="29d99320-f1a9-4178-97f6-dc0a088aec0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 21:12:33.553792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:33.547166 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:12:33.700591 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:33.700081 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body= Oct 30 21:12:33.700591 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:33.700142 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" podUID="29d99320-f1a9-4178-97f6-dc0a088aec0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" Oct 30 21:12:34.697381 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:34.697346 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:12:34.698808 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:34.698760 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:12:52.512569 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:12:52.512533 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:12:53.166046 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.166008 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs"] Oct 30 21:12:53.169867 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.169847 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.175021 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.175005 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d95261-openshift-default-dockercfg-qrtlx\"" Oct 30 21:12:53.247199 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.247172 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-istio-envoy\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.247381 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.247358 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c705c9c-fef2-422b-b539-4007e28fedee-istio-token\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.247486 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.247455 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-istio-data\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.247542 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.247524 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c705c9c-fef2-422b-b539-4007e28fedee-istio-podinfo\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.247601 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.247556 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xj9q\" (UniqueName: \"kubernetes.io/projected/5c705c9c-fef2-422b-b539-4007e28fedee-kube-api-access-9xj9q\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.247653 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.247593 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-workload-socket\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.247653 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.247633 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c705c9c-fef2-422b-b539-4007e28fedee-istiod-ca-cert\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.247759 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.247656 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-credential-socket\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.247859 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.247835 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-workload-certs\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.270550 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.270520 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs"] Oct 30 21:12:53.349541 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.349502 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c705c9c-fef2-422b-b539-4007e28fedee-istio-podinfo\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.349748 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.349557 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xj9q\" (UniqueName: \"kubernetes.io/projected/5c705c9c-fef2-422b-b539-4007e28fedee-kube-api-access-9xj9q\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.349748 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.349606 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-workload-socket\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.349748 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.349653 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c705c9c-fef2-422b-b539-4007e28fedee-istiod-ca-cert\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.349748 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.349710 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-credential-socket\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.350181 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.349824 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-workload-certs\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.350181 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.349940 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-istio-envoy\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.350181 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.350146 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c705c9c-fef2-422b-b539-4007e28fedee-istio-token\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.350361 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.350263 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-istio-data\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.350520 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.350499 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-workload-certs\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.350650 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.350625 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c705c9c-fef2-422b-b539-4007e28fedee-istiod-ca-cert\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.351100 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.351078 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-credential-socket\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.351237 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.351191 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-workload-socket\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.357780 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.357718 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c705c9c-fef2-422b-b539-4007e28fedee-istio-podinfo\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.357995 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.357971 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-istio-data\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.358298 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.358256 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-istio-envoy\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.361906 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.361860 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c705c9c-fef2-422b-b539-4007e28fedee-istio-token\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.362617 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.362593 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xj9q\" (UniqueName: \"kubernetes.io/projected/5c705c9c-fef2-422b-b539-4007e28fedee-kube-api-access-9xj9q\") pod \"e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:53.507801 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:53.497541 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:54.140356 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.138128 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z"] Oct 30 21:12:54.196037 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.196003 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z"] Oct 30 21:12:54.196221 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.196067 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs"] Oct 30 21:12:54.196221 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.196179 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.207662 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.207619 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:12:54.207797 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.207711 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:12:54.207868 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.207799 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:12:54.238978 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.238956 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e1e0f5-openshift-default-dockercfg-jmrsc\"" Oct 30 21:12:54.263998 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.263967 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092690bd-ffd6-4893-96b0-025b55539fd7-istio-podinfo\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.264141 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.264017 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-workload-certs\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.264141 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.264048 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092690bd-ffd6-4893-96b0-025b55539fd7-istiod-ca-cert\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.264275 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.264183 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-istio-envoy\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.264275 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.264255 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-workload-socket\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.264400 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.264281 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092690bd-ffd6-4893-96b0-025b55539fd7-istio-token\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.264464 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.264444 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjpj6\" (UniqueName: \"kubernetes.io/projected/092690bd-ffd6-4893-96b0-025b55539fd7-kube-api-access-bjpj6\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.264526 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.264467 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-credential-socket\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.264624 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.264607 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-istio-data\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.366420 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.366388 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-workload-socket\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.366420 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.366432 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092690bd-ffd6-4893-96b0-025b55539fd7-istio-token\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.366691 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.366531 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjpj6\" (UniqueName: \"kubernetes.io/projected/092690bd-ffd6-4893-96b0-025b55539fd7-kube-api-access-bjpj6\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.366691 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.366552 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-credential-socket\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.366691 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.366624 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-istio-data\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.366691 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.366655 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092690bd-ffd6-4893-96b0-025b55539fd7-istio-podinfo\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.366691 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.366683 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-workload-certs\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.366990 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.366708 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092690bd-ffd6-4893-96b0-025b55539fd7-istiod-ca-cert\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.366990 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.366780 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-istio-envoy\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.367213 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.367190 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-istio-data\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.367446 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.367427 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-credential-socket\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.367699 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.367672 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-workload-certs\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.368045 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.368018 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-workload-socket\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.368417 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.368395 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092690bd-ffd6-4893-96b0-025b55539fd7-istiod-ca-cert\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.369587 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.369560 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-istio-envoy\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.371069 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.371049 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092690bd-ffd6-4893-96b0-025b55539fd7-istio-podinfo\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.376236 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.376212 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092690bd-ffd6-4893-96b0-025b55539fd7-istio-token\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.377085 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.377061 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjpj6\" (UniqueName: \"kubernetes.io/projected/092690bd-ffd6-4893-96b0-025b55539fd7-kube-api-access-bjpj6\") pod \"e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.517679 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.517643 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:54.815835 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.815807 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z"] Oct 30 21:12:54.816223 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:12:54.816201 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092690bd_ffd6_4893_96b0_025b55539fd7.slice/crio-775d69ab066522f7cdadffece867dcb819008440e4901107f60c629a95ee80eb WatchSource:0}: Error finding container 775d69ab066522f7cdadffece867dcb819008440e4901107f60c629a95ee80eb: Status 404 returned error can't find the container with id 775d69ab066522f7cdadffece867dcb819008440e4901107f60c629a95ee80eb Oct 30 21:12:54.819937 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.819899 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:12:54.820041 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.819992 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:12:54.820041 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:54.820037 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:12:55.037565 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:55.037526 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" event={"ID":"5c705c9c-fef2-422b-b539-4007e28fedee","Type":"ContainerStarted","Data":"99739c29e97d34cb7dae40b0820ebf495474db9637a5116103479f887398b8ce"} Oct 30 21:12:55.038106 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:55.037592 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" event={"ID":"5c705c9c-fef2-422b-b539-4007e28fedee","Type":"ContainerStarted","Data":"3b0b00ed351e7c6a8e86444dd27c5920270a3dcb68447c1b6b753a98d80a7396"} Oct 30 21:12:55.039857 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:55.039826 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" event={"ID":"092690bd-ffd6-4893-96b0-025b55539fd7","Type":"ContainerStarted","Data":"e4402d746afd44d6be7b8f1a08df02ff2a8b238243c108353463b54255f4db2c"} Oct 30 21:12:55.039987 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:55.039882 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" event={"ID":"092690bd-ffd6-4893-96b0-025b55539fd7","Type":"ContainerStarted","Data":"775d69ab066522f7cdadffece867dcb819008440e4901107f60c629a95ee80eb"} Oct 30 21:12:55.059894 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:55.059848 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" podStartSLOduration=2.059835442 podStartE2EDuration="2.059835442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:55.057861406 +0000 UTC m=+2080.237757912" watchObservedRunningTime="2026-10-30 21:12:55.059835442 +0000 UTC m=+2080.239732006" Oct 30 21:12:55.083812 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:55.083680 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" podStartSLOduration=2.083661692 podStartE2EDuration="2.083661692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:55.082866475 +0000 UTC m=+2080.262762982" watchObservedRunningTime="2026-10-30 21:12:55.083661692 +0000 UTC m=+2080.263558199" Oct 30 21:12:55.514788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:55.513709 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:12:55.514788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:55.513872 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 21:12:55.514788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:55.513919 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" podUID="5c705c9c-fef2-422b-b539-4007e28fedee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 21:12:55.527805 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:55.520442 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:12:55.527805 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:55.520608 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 21:12:55.527805 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:55.520649 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" podUID="092690bd-ffd6-4893-96b0-025b55539fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 21:12:56.525485 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:56.524303 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 21:12:56.525485 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:56.524375 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" podUID="092690bd-ffd6-4893-96b0-025b55539fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 21:12:56.556841 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:56.547382 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 21:12:56.556841 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:56.547442 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" podUID="5c705c9c-fef2-422b-b539-4007e28fedee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 21:12:57.517651 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:57.517612 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 21:12:57.518094 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:57.517963 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" podUID="5c705c9c-fef2-422b-b539-4007e28fedee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 21:12:57.518948 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:57.518836 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 21:12:57.519128 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:57.519105 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" podUID="092690bd-ffd6-4893-96b0-025b55539fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 21:12:58.524792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:58.522258 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 21:12:58.524792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:58.522338 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" podUID="092690bd-ffd6-4893-96b0-025b55539fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 21:12:58.524792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:58.522841 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 21:12:58.524792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:58.522895 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" podUID="5c705c9c-fef2-422b-b539-4007e28fedee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 21:12:59.511790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:59.511150 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 21:12:59.511790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:59.511213 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" podUID="5c705c9c-fef2-422b-b539-4007e28fedee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 21:12:59.526789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:59.523190 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 21:12:59.526789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:12:59.523236 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" podUID="092690bd-ffd6-4893-96b0-025b55539fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 21:13:00.534790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:13:00.526633 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 21:13:00.534790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:13:00.526694 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" podUID="5c705c9c-fef2-422b-b539-4007e28fedee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 21:13:00.534790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:13:00.527103 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 21:13:00.534790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:13:00.527154 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" podUID="092690bd-ffd6-4893-96b0-025b55539fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 21:13:01.514313 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:13:01.514282 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:13:01.514761 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:13:01.514745 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:13:01.515464 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:13:01.515441 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:13:01.523429 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:13:01.523408 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:13:01.523675 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:13:01.523650 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:13:01.525224 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:13:01.525196 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:13:16.347252 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:13:16.347170 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 21:13:16.349807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:13:16.349784 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 21:13:22.536401 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:13:22.536355 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:13:52.512642 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:13:52.512605 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:14:07.378256 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:07.378217 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb"] Oct 30 21:14:07.378918 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:07.378453 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" podUID="91501d35-e8f3-4a20-93b6-6b2bda59e4e4" containerName="istio-proxy" containerID="cri-o://5cb33479546f22b38e72b41c8aa93977f768a806674c4c6c9b2ee8d746d33c08" gracePeriod=30 Oct 30 21:14:12.663501 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.663427 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:14:12.765417 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.765374 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-podinfo\") pod \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " Oct 30 21:14:12.765650 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.765564 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-data\") pod \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " Oct 30 21:14:12.765719 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.765680 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-workload-certs\") pod \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " Oct 30 21:14:12.765843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.765820 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-credential-socket\") pod \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " Oct 30 21:14:12.765951 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.765885 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istiod-ca-cert\") pod \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " Oct 30 21:14:12.766015 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.765937 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-data" pod "91501d35-e8f3-4a20-93b6-6b2bda59e4e4" (UID: "91501d35-e8f3-4a20-93b6-6b2bda59e4e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:12.766015 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.765952 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-kube-api-access-czqsj\" (UniqueName: \"kubernetes.io/projected/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-kube-api-access-czqsj\") pod \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " Oct 30 21:14:12.766170 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.766113 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-workload-certs" pod "91501d35-e8f3-4a20-93b6-6b2bda59e4e4" (UID: "91501d35-e8f3-4a20-93b6-6b2bda59e4e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:12.766170 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.766116 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-token\" (UniqueName: \"kubernetes.io/projected/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-token\") pod \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " Oct 30 21:14:12.766287 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.766258 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-workload-socket\") pod \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " Oct 30 21:14:12.767360 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.766340 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-envoy\") pod \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\" (UID: \"91501d35-e8f3-4a20-93b6-6b2bda59e4e4\") " Oct 30 21:14:12.767360 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.766567 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-credential-socket" pod "91501d35-e8f3-4a20-93b6-6b2bda59e4e4" (UID: "91501d35-e8f3-4a20-93b6-6b2bda59e4e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:12.767360 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.766940 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istiod-ca-cert" pod "91501d35-e8f3-4a20-93b6-6b2bda59e4e4" (UID: "91501d35-e8f3-4a20-93b6-6b2bda59e4e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:12.767360 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.767195 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-workload-socket" pod "91501d35-e8f3-4a20-93b6-6b2bda59e4e4" (UID: "91501d35-e8f3-4a20-93b6-6b2bda59e4e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:12.767360 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.767262 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:12.767360 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.767283 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:12.767360 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.767300 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:12.767360 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.767314 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:12.768451 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.768413 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-podinfo" pod "91501d35-e8f3-4a20-93b6-6b2bda59e4e4" (UID: "91501d35-e8f3-4a20-93b6-6b2bda59e4e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:14:12.768611 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.768551 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-token" pod "91501d35-e8f3-4a20-93b6-6b2bda59e4e4" (UID: "91501d35-e8f3-4a20-93b6-6b2bda59e4e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:12.768798 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.768746 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-kube-api-access-czqsj" pod "91501d35-e8f3-4a20-93b6-6b2bda59e4e4" (UID: "91501d35-e8f3-4a20-93b6-6b2bda59e4e4"). InnerVolumeSpecName "kube-api-access-czqsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:12.769477 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.769454 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-envoy" pod "91501d35-e8f3-4a20-93b6-6b2bda59e4e4" (UID: "91501d35-e8f3-4a20-93b6-6b2bda59e4e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:14:12.874518 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.874396 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czqsj\" (UniqueName: \"kubernetes.io/projected/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-kube-api-access-czqsj\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:12.874518 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.874430 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:12.874518 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.874443 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:12.874518 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.874458 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:12.874518 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:12.874469 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91501d35-e8f3-4a20-93b6-6b2bda59e4e4-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:14:13.204282 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:13.204194 4848 generic.go:356] "Generic (PLEG): container finished" podID="91501d35-e8f3-4a20-93b6-6b2bda59e4e4" containerID="5cb33479546f22b38e72b41c8aa93977f768a806674c4c6c9b2ee8d746d33c08" exitCode=0 Oct 30 21:14:13.204282 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:13.204242 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" Oct 30 21:14:13.204499 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:13.204279 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" event={"ID":"91501d35-e8f3-4a20-93b6-6b2bda59e4e4","Type":"ContainerDied","Data":"5cb33479546f22b38e72b41c8aa93977f768a806674c4c6c9b2ee8d746d33c08"} Oct 30 21:14:13.204499 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:13.204360 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb" event={"ID":"91501d35-e8f3-4a20-93b6-6b2bda59e4e4","Type":"ContainerDied","Data":"4d8b8bd23e6d3d853cd7c2b9840c45120d1388cc7052604770c6a6af935afafa"} Oct 30 21:14:13.204499 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:13.204386 4848 scope.go:122] "RemoveContainer" containerID="5cb33479546f22b38e72b41c8aa93977f768a806674c4c6c9b2ee8d746d33c08" Oct 30 21:14:13.215505 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:13.215483 4848 scope.go:122] "RemoveContainer" containerID="5cb33479546f22b38e72b41c8aa93977f768a806674c4c6c9b2ee8d746d33c08" Oct 30 21:14:13.215830 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:14:13.215805 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb33479546f22b38e72b41c8aa93977f768a806674c4c6c9b2ee8d746d33c08\": container with ID starting with 5cb33479546f22b38e72b41c8aa93977f768a806674c4c6c9b2ee8d746d33c08 not found: ID does not exist" containerID="5cb33479546f22b38e72b41c8aa93977f768a806674c4c6c9b2ee8d746d33c08" Oct 30 21:14:13.215944 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:13.215837 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb33479546f22b38e72b41c8aa93977f768a806674c4c6c9b2ee8d746d33c08"} err="failed to get container status \"5cb33479546f22b38e72b41c8aa93977f768a806674c4c6c9b2ee8d746d33c08\": rpc error: code = NotFound desc = could not find container \"5cb33479546f22b38e72b41c8aa93977f768a806674c4c6c9b2ee8d746d33c08\": container with ID starting with 5cb33479546f22b38e72b41c8aa93977f768a806674c4c6c9b2ee8d746d33c08 not found: ID does not exist" Oct 30 21:14:13.291361 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:13.291317 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb"] Oct 30 21:14:13.299275 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:13.299245 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-m4zkb"] Oct 30 21:14:14.934901 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:14.934872 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91501d35-e8f3-4a20-93b6-6b2bda59e4e4" path="/var/lib/kubelet/pods/91501d35-e8f3-4a20-93b6-6b2bda59e4e4/volumes" Oct 30 21:14:17.487390 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:14:17.487351 4848 scope.go:122] "RemoveContainer" containerID="486ca2cf193f5a0fbacdb3d1260439c213c223c2770ffaf2340ec3d859d10c87" Oct 30 21:14:22.513052 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:14:22.513013 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:14:52.515501 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:14:52.515414 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:15:06.308628 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.308586 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8"] Oct 30 21:15:06.309426 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.309407 4848 cpu_manager.go:414] "containerMap: removing container" podUID="91501d35-e8f3-4a20-93b6-6b2bda59e4e4" containerName="istio-proxy" Oct 30 21:15:06.309484 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.309428 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91501d35-e8f3-4a20-93b6-6b2bda59e4e4" containerName="istio-proxy" Oct 30 21:15:06.309553 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.309544 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="91501d35-e8f3-4a20-93b6-6b2bda59e4e4" containerName="istio-proxy" Oct 30 21:15:06.313430 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.313412 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.323281 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.323262 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-345bc4-openshift-default-dockercfg-fck6v\"" Oct 30 21:15:06.335271 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.335239 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8"] Oct 30 21:15:06.432756 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.432723 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a829dfbb-7c3e-4c29-8644-87a3d2269978-istiod-ca-cert\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.432965 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.432798 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-envoy\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.432965 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.432877 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-workload-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.433049 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.433011 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-data\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.433090 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.433057 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45mcg\" (UniqueName: \"kubernetes.io/projected/a829dfbb-7c3e-4c29-8644-87a3d2269978-kube-api-access-45mcg\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.433180 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.433159 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-workload-certs\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.433261 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.433247 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-credential-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.433298 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.433272 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-podinfo\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.433442 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.433425 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-token\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.535133 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.535093 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-data\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.535327 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.535143 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45mcg\" (UniqueName: \"kubernetes.io/projected/a829dfbb-7c3e-4c29-8644-87a3d2269978-kube-api-access-45mcg\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.535327 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.535187 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-workload-certs\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.535327 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.535216 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-credential-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.535327 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.535235 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-podinfo\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.535595 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.535369 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-token\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.535595 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.535430 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a829dfbb-7c3e-4c29-8644-87a3d2269978-istiod-ca-cert\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.535595 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.535485 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-envoy\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.535595 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.535522 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-workload-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.535595 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.535566 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-data\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.535902 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.535686 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-workload-certs\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.535902 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.535748 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-credential-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.536210 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.536184 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a829dfbb-7c3e-4c29-8644-87a3d2269978-istiod-ca-cert\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.536400 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.536381 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-workload-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.537819 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.537755 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-podinfo\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.537819 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.537800 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-envoy\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.547277 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.547252 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-token\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.547398 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.547324 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45mcg\" (UniqueName: \"kubernetes.io/projected/a829dfbb-7c3e-4c29-8644-87a3d2269978-kube-api-access-45mcg\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.628264 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.628181 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:06.982791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.963832 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:06.982791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.963922 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:06.982791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:06.963962 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:07.021715 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:07.021685 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8"] Oct 30 21:15:07.902523 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:07.902485 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" event={"ID":"a829dfbb-7c3e-4c29-8644-87a3d2269978","Type":"ContainerStarted","Data":"b524d5aeb0703dfd01ef73abc9e9eb6407926dbf65809d01ffbdeae7f92c610c"} Oct 30 21:15:07.902948 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:07.902565 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" event={"ID":"a829dfbb-7c3e-4c29-8644-87a3d2269978","Type":"ContainerStarted","Data":"7676e295e0a069cc288180c0ba05c39d11be40dbf4eef682ae0850a08a4bf02b"} Oct 30 21:15:07.939247 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:07.939172 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" podStartSLOduration=1.939154217 podStartE2EDuration="1.939154217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:07.937383232 +0000 UTC m=+2213.117279741" watchObservedRunningTime="2026-10-30 21:15:07.939154217 +0000 UTC m=+2213.119050725" Oct 30 21:15:08.629433 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:08.629354 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:08.632374 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:08.632320 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 21:15:08.632684 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:08.632637 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" podUID="a829dfbb-7c3e-4c29-8644-87a3d2269978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 21:15:09.637605 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:09.637561 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 21:15:09.638312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:09.638280 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" podUID="a829dfbb-7c3e-4c29-8644-87a3d2269978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 21:15:10.629201 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:10.629162 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 21:15:10.629393 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:10.629242 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" podUID="a829dfbb-7c3e-4c29-8644-87a3d2269978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 21:15:11.630274 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:11.630231 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 21:15:11.631041 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:11.631005 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" podUID="a829dfbb-7c3e-4c29-8644-87a3d2269978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 21:15:12.633788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:12.629172 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 21:15:12.633788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:12.629232 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" podUID="a829dfbb-7c3e-4c29-8644-87a3d2269978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 21:15:13.629787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:13.629261 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 21:15:13.629787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:13.629340 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" podUID="a829dfbb-7c3e-4c29-8644-87a3d2269978" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 21:15:14.634042 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:14.634004 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:14.634466 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:14.634380 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:14.635167 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:14.635150 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:15:21.336806 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.336756 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk"] Oct 30 21:15:21.341860 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.341836 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.361373 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.361343 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk"] Oct 30 21:15:21.407183 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.407153 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h"] Oct 30 21:15:21.411726 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.411697 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.448199 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.448140 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h"] Oct 30 21:15:21.449963 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.449932 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-workload-certs\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.450200 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.450172 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-podinfo\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.450322 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.450215 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-data\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.450322 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.450261 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-data\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.450436 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.450324 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-token\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.450436 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.450381 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-credential-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.450436 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.450417 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-credential-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.450593 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.450461 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istiod-ca-cert\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.450593 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.450549 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-workload-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.450694 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.450612 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7wng\" (UniqueName: \"kubernetes.io/projected/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-kube-api-access-k7wng\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.450694 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.450666 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78a62f4f-ca51-47f0-8c4a-9d146a810826-istiod-ca-cert\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.450809 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.450747 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpvtx\" (UniqueName: \"kubernetes.io/projected/78a62f4f-ca51-47f0-8c4a-9d146a810826-kube-api-access-tpvtx\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.450868 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.450803 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-workload-certs\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.450920 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.450864 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-envoy\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.450975 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.450938 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-token\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.451030 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.450977 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-podinfo\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.451030 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.451005 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-workload-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.451132 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.451032 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-envoy\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.552607 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.552571 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-workload-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.552852 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.552631 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7wng\" (UniqueName: \"kubernetes.io/projected/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-kube-api-access-k7wng\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.552852 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.552674 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78a62f4f-ca51-47f0-8c4a-9d146a810826-istiod-ca-cert\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.552852 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.552741 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpvtx\" (UniqueName: \"kubernetes.io/projected/78a62f4f-ca51-47f0-8c4a-9d146a810826-kube-api-access-tpvtx\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.553110 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.553082 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-workload-certs\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.553362 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.553165 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-envoy\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.553362 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.553223 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-workload-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.553362 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.553334 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-token\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.553656 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.553372 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-podinfo\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.553656 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.553401 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-workload-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.553656 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.553420 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-workload-certs\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.553656 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.553426 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-envoy\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.553656 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.553545 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78a62f4f-ca51-47f0-8c4a-9d146a810826-istiod-ca-cert\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.553656 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.553561 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-workload-certs\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.553656 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.553590 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-podinfo\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.553656 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.553616 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-data\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.553656 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.553655 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-data\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.554404 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.553723 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-token\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.554404 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.553840 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-credential-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.554404 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.553877 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-credential-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.554404 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.553932 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istiod-ca-cert\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.554726 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.554699 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-data\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.554726 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.554704 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istiod-ca-cert\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.555024 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.555000 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-credential-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.555106 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.555010 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-data\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.555277 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.555254 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-workload-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.555347 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.555320 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-credential-socket\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.555527 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.555504 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-workload-certs\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.556723 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.556696 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-envoy\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.556965 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.556943 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-podinfo\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.557553 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.557527 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-envoy\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.559490 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.559461 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-podinfo\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.563005 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.562962 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpvtx\" (UniqueName: \"kubernetes.io/projected/78a62f4f-ca51-47f0-8c4a-9d146a810826-kube-api-access-tpvtx\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.563111 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.563090 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7wng\" (UniqueName: \"kubernetes.io/projected/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-kube-api-access-k7wng\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.564075 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.564056 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-token\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.566799 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.566739 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-token\") pod \"e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.657578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.657501 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:21.747885 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.747710 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:21.952040 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.952010 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk"] Oct 30 21:15:21.952356 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:15:21.952324 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03f9217_ab3a_4aac_a1ea_a04deeb81d36.slice/crio-20d87c6f5c39933ac57093f9f0f5c7134168734863080ce047d7ad06a0d985a7 WatchSource:0}: Error finding container 20d87c6f5c39933ac57093f9f0f5c7134168734863080ce047d7ad06a0d985a7: Status 404 returned error can't find the container with id 20d87c6f5c39933ac57093f9f0f5c7134168734863080ce047d7ad06a0d985a7 Oct 30 21:15:21.955705 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.955668 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:21.955840 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.955759 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:21.955840 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:21.955819 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:22.081705 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:15:22.081671 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a62f4f_ca51_47f0_8c4a_9d146a810826.slice/crio-c136f43b861b1426c9eaccbc0b55f308da62ef43cab3e877ebfc61327a1ffc8f WatchSource:0}: Error finding container c136f43b861b1426c9eaccbc0b55f308da62ef43cab3e877ebfc61327a1ffc8f: Status 404 returned error can't find the container with id c136f43b861b1426c9eaccbc0b55f308da62ef43cab3e877ebfc61327a1ffc8f Oct 30 21:15:22.081917 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:22.081875 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h"] Oct 30 21:15:22.085148 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:22.085107 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:22.085261 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:22.085199 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:22.085309 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:22.085294 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:22.246848 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:22.246714 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" event={"ID":"78a62f4f-ca51-47f0-8c4a-9d146a810826","Type":"ContainerStarted","Data":"c136f43b861b1426c9eaccbc0b55f308da62ef43cab3e877ebfc61327a1ffc8f"} Oct 30 21:15:22.249682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:22.249649 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" event={"ID":"c03f9217-ab3a-4aac-a1ea-a04deeb81d36","Type":"ContainerStarted","Data":"8ade5cd7f2124260dd5f51fa0b58e8039bfacdaea3f3bc7f2195f8b4738e37f1"} Oct 30 21:15:22.249897 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:22.249708 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" event={"ID":"c03f9217-ab3a-4aac-a1ea-a04deeb81d36","Type":"ContainerStarted","Data":"20d87c6f5c39933ac57093f9f0f5c7134168734863080ce047d7ad06a0d985a7"} Oct 30 21:15:22.285018 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:22.284969 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" podStartSLOduration=1.284956666 podStartE2EDuration="1.284956666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:22.282283641 +0000 UTC m=+2227.462180159" watchObservedRunningTime="2026-10-30 21:15:22.284956666 +0000 UTC m=+2227.464853172" Oct 30 21:15:22.525245 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:15:22.525143 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:15:22.658608 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:22.658577 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:23.271598 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:23.271552 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" event={"ID":"78a62f4f-ca51-47f0-8c4a-9d146a810826","Type":"ContainerStarted","Data":"2b89067dd5537d449efcf2fc389b30fcf11f1935f43635f10869cbfb4c9eab29"} Oct 30 21:15:23.301734 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:23.301680 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" podStartSLOduration=2.301665369 podStartE2EDuration="2.301665369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:23.294878662 +0000 UTC m=+2228.474775170" watchObservedRunningTime="2026-10-30 21:15:23.301665369 +0000 UTC m=+2228.481561950" Oct 30 21:15:23.660194 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:23.659883 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 21:15:23.660194 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:23.659959 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" podUID="c03f9217-ab3a-4aac-a1ea-a04deeb81d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 21:15:23.764265 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:23.764230 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:23.766240 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:23.766202 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 30 21:15:23.766390 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:23.766261 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" podUID="78a62f4f-ca51-47f0-8c4a-9d146a810826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 30 21:15:24.658486 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:24.658448 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 21:15:24.658804 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:24.658739 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" podUID="c03f9217-ab3a-4aac-a1ea-a04deeb81d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 21:15:24.800573 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:24.800538 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 30 21:15:24.801186 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:24.801151 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" podUID="78a62f4f-ca51-47f0-8c4a-9d146a810826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 30 21:15:25.658359 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:25.658325 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 21:15:25.658613 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:25.658583 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" podUID="c03f9217-ab3a-4aac-a1ea-a04deeb81d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 21:15:25.754791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:25.754197 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 30 21:15:25.754791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:25.754263 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" podUID="78a62f4f-ca51-47f0-8c4a-9d146a810826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 30 21:15:26.658992 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:26.658954 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 21:15:26.659591 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:26.659554 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" podUID="c03f9217-ab3a-4aac-a1ea-a04deeb81d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 21:15:26.752030 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:26.751539 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 30 21:15:26.752030 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:26.751628 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" podUID="78a62f4f-ca51-47f0-8c4a-9d146a810826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 30 21:15:27.658270 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:27.658236 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 21:15:27.658500 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:27.658480 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" podUID="c03f9217-ab3a-4aac-a1ea-a04deeb81d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 21:15:27.753832 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:27.751114 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 30 21:15:27.753832 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:27.751187 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" podUID="78a62f4f-ca51-47f0-8c4a-9d146a810826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 30 21:15:28.658311 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:28.658269 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" start-of-body= Oct 30 21:15:28.658518 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:28.658346 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" podUID="c03f9217-ab3a-4aac-a1ea-a04deeb81d36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.230:15021/healthz/ready\": dial tcp 10.134.0.230:15021: connect: connection refused" Oct 30 21:15:28.749463 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:28.749409 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" start-of-body= Oct 30 21:15:28.749664 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:28.749487 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" podUID="78a62f4f-ca51-47f0-8c4a-9d146a810826" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.231:15021/healthz/ready\": dial tcp 10.134.0.231:15021: connect: connection refused" Oct 30 21:15:29.662898 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:29.662866 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:29.663409 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:29.663231 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:29.663950 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:29.663928 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:15:29.756606 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:29.756571 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:29.757133 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:29.757107 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:29.758279 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:15:29.758258 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:15:52.517581 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:15:52.517545 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:16:22.599183 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:16:22.599084 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:16:39.000936 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:16:39.000554 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:52.654694 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:16:52.654651 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:17:09.919293 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:09.919257 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6"] Oct 30 21:17:09.924493 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:09.924470 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:09.941789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:09.941745 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6"] Oct 30 21:17:10.061659 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.061623 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.061879 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.061670 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.061957 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.061911 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.062040 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.062013 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4htf\" (UniqueName: \"kubernetes.io/projected/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-kube-api-access-g4htf\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.062174 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.062158 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.062243 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.062201 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.062337 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.062320 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.062565 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.062543 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.062676 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.062667 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.163978 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.163944 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.164189 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.163989 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.164189 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.164044 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.164189 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.164076 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4htf\" (UniqueName: \"kubernetes.io/projected/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-kube-api-access-g4htf\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.164396 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.164216 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.164396 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.164273 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.164396 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.164335 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.164633 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.164441 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.164633 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.164468 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.164633 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.164488 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.164633 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.164546 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.164633 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.164602 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.165046 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.165024 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.165710 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.165115 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.166888 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.166863 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.167077 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.167057 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.180185 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.180165 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4htf\" (UniqueName: \"kubernetes.io/projected/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-kube-api-access-g4htf\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.180311 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.180256 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-qbpx6\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.241566 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.241529 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:10.576483 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.576453 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6"] Oct 30 21:17:10.576647 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:17:10.576594 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e35d342_d32b_4cbb_8249_6a4c12ce82cd.slice/crio-04445245a5cfef6d346b6ad08890d48be57e8dec27c1ce4a9a502b2e3cf04f74 WatchSource:0}: Error finding container 04445245a5cfef6d346b6ad08890d48be57e8dec27c1ce4a9a502b2e3cf04f74: Status 404 returned error can't find the container with id 04445245a5cfef6d346b6ad08890d48be57e8dec27c1ce4a9a502b2e3cf04f74 Oct 30 21:17:10.579760 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.579720 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:17:10.579894 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.579809 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:17:10.579894 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.579839 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:17:10.869712 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.869608 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" event={"ID":"9e35d342-d32b-4cbb-8249-6a4c12ce82cd","Type":"ContainerStarted","Data":"9ab7378278af6a7d204ba1663483f335916eca23b93912d2cb3aeb55b8ae367d"} Oct 30 21:17:10.869936 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.869719 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" event={"ID":"9e35d342-d32b-4cbb-8249-6a4c12ce82cd","Type":"ContainerStarted","Data":"04445245a5cfef6d346b6ad08890d48be57e8dec27c1ce4a9a502b2e3cf04f74"} Oct 30 21:17:10.930308 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:10.930252 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" podStartSLOduration=1.930235684 podStartE2EDuration="1.930235684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:17:10.925199712 +0000 UTC m=+2336.105096237" watchObservedRunningTime="2026-10-30 21:17:10.930235684 +0000 UTC m=+2336.110132195" Oct 30 21:17:11.242586 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:11.242545 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:11.245244 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:11.245209 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-qbpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 21:17:11.245415 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:11.245289 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" podUID="9e35d342-d32b-4cbb-8249-6a4c12ce82cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 21:17:12.242165 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:12.242127 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-qbpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 21:17:12.242561 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:12.242182 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" podUID="9e35d342-d32b-4cbb-8249-6a4c12ce82cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 21:17:13.243894 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:13.243856 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-qbpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 21:17:13.244481 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:13.244448 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" podUID="9e35d342-d32b-4cbb-8249-6a4c12ce82cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 21:17:13.995946 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:13.995897 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:17:14.243533 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:14.243082 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-qbpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 21:17:14.243533 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:14.243145 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" podUID="9e35d342-d32b-4cbb-8249-6a4c12ce82cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 21:17:15.244790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:15.244224 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-qbpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 21:17:15.244790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:15.244292 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" podUID="9e35d342-d32b-4cbb-8249-6a4c12ce82cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 21:17:16.247801 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:16.247272 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-qbpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 21:17:16.247801 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:16.247370 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" podUID="9e35d342-d32b-4cbb-8249-6a4c12ce82cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 21:17:17.251633 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:17.249146 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-qbpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 21:17:17.251633 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:17.249224 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" podUID="9e35d342-d32b-4cbb-8249-6a4c12ce82cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 21:17:18.242752 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:18.242709 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-qbpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 21:17:18.242985 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:18.242810 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" podUID="9e35d342-d32b-4cbb-8249-6a4c12ce82cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 21:17:19.242450 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:19.242412 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-qbpx6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" start-of-body= Oct 30 21:17:19.242936 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:19.242482 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" podUID="9e35d342-d32b-4cbb-8249-6a4c12ce82cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.232:15021/healthz/ready\": dial tcp 10.134.0.232:15021: connect: connection refused" Oct 30 21:17:20.865576 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:20.865523 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:20.867408 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:20.866851 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:20.869018 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:17:20.868994 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:17:22.520458 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:17:22.520415 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:17:52.514673 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:17:52.514584 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:18:16.400991 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:18:16.400957 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 21:18:16.405046 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:18:16.405023 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 21:18:22.512553 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:18:22.512513 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:18:52.514501 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:18:52.514458 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:19:22.512485 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:19:22.512406 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:19:52.648790 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:19:52.647963 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:20:22.514164 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:20:22.514128 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:20:52.512968 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:20:52.512887 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:21:22.512656 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:21:22.512617 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:21:52.572805 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:21:52.569590 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:22:06.992832 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:06.992796 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8"] Oct 30 21:22:06.993310 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:06.993126 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" podUID="a829dfbb-7c3e-4c29-8644-87a3d2269978" containerName="istio-proxy" containerID="cri-o://b524d5aeb0703dfd01ef73abc9e9eb6407926dbf65809d01ffbdeae7f92c610c" gracePeriod=30 Oct 30 21:22:12.153291 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.153251 4848 generic.go:356] "Generic (PLEG): container finished" podID="a829dfbb-7c3e-4c29-8644-87a3d2269978" containerID="b524d5aeb0703dfd01ef73abc9e9eb6407926dbf65809d01ffbdeae7f92c610c" exitCode=0 Oct 30 21:22:12.153757 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.153397 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" event={"ID":"a829dfbb-7c3e-4c29-8644-87a3d2269978","Type":"ContainerDied","Data":"b524d5aeb0703dfd01ef73abc9e9eb6407926dbf65809d01ffbdeae7f92c610c"} Oct 30 21:22:12.382402 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.382374 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:22:12.443846 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.443810 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-data\") pod \"a829dfbb-7c3e-4c29-8644-87a3d2269978\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " Oct 30 21:22:12.444059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.443881 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-credential-socket\") pod \"a829dfbb-7c3e-4c29-8644-87a3d2269978\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " Oct 30 21:22:12.444059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.443968 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-workload-socket\") pod \"a829dfbb-7c3e-4c29-8644-87a3d2269978\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " Oct 30 21:22:12.444059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.444018 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-token\" (UniqueName: \"kubernetes.io/projected/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-token\") pod \"a829dfbb-7c3e-4c29-8644-87a3d2269978\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " Oct 30 21:22:12.444232 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.444056 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a829dfbb-7c3e-4c29-8644-87a3d2269978-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a829dfbb-7c3e-4c29-8644-87a3d2269978-istiod-ca-cert\") pod \"a829dfbb-7c3e-4c29-8644-87a3d2269978\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " Oct 30 21:22:12.444232 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.444109 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-data" pod "a829dfbb-7c3e-4c29-8644-87a3d2269978" (UID: "a829dfbb-7c3e-4c29-8644-87a3d2269978"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:22:12.444232 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.444142 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-podinfo\") pod \"a829dfbb-7c3e-4c29-8644-87a3d2269978\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " Oct 30 21:22:12.444232 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.444180 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-workload-certs\") pod \"a829dfbb-7c3e-4c29-8644-87a3d2269978\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " Oct 30 21:22:12.444441 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.444234 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-envoy\") pod \"a829dfbb-7c3e-4c29-8644-87a3d2269978\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " Oct 30 21:22:12.444441 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.444270 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-workload-socket" pod "a829dfbb-7c3e-4c29-8644-87a3d2269978" (UID: "a829dfbb-7c3e-4c29-8644-87a3d2269978"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:22:12.444554 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.444437 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-credential-socket" pod "a829dfbb-7c3e-4c29-8644-87a3d2269978" (UID: "a829dfbb-7c3e-4c29-8644-87a3d2269978"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:22:12.444554 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.444453 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a829dfbb-7c3e-4c29-8644-87a3d2269978-kube-api-access-45mcg\" (UniqueName: \"kubernetes.io/projected/a829dfbb-7c3e-4c29-8644-87a3d2269978-kube-api-access-45mcg\") pod \"a829dfbb-7c3e-4c29-8644-87a3d2269978\" (UID: \"a829dfbb-7c3e-4c29-8644-87a3d2269978\") " Oct 30 21:22:12.445084 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.445052 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-workload-certs" pod "a829dfbb-7c3e-4c29-8644-87a3d2269978" (UID: "a829dfbb-7c3e-4c29-8644-87a3d2269978"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:22:12.445460 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.445437 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:12.445620 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.445602 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:12.445724 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.445711 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:12.445847 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.445831 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:12.445969 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.445759 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a829dfbb-7c3e-4c29-8644-87a3d2269978-istiod-ca-cert" pod "a829dfbb-7c3e-4c29-8644-87a3d2269978" (UID: "a829dfbb-7c3e-4c29-8644-87a3d2269978"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:22:12.448817 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.448728 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a829dfbb-7c3e-4c29-8644-87a3d2269978-kube-api-access-45mcg" pod "a829dfbb-7c3e-4c29-8644-87a3d2269978" (UID: "a829dfbb-7c3e-4c29-8644-87a3d2269978"). InnerVolumeSpecName "kube-api-access-45mcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:22:12.448996 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.448973 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-token" pod "a829dfbb-7c3e-4c29-8644-87a3d2269978" (UID: "a829dfbb-7c3e-4c29-8644-87a3d2269978"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:22:12.450297 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.450272 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-podinfo" pod "a829dfbb-7c3e-4c29-8644-87a3d2269978" (UID: "a829dfbb-7c3e-4c29-8644-87a3d2269978"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:22:12.450297 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.450280 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-envoy" pod "a829dfbb-7c3e-4c29-8644-87a3d2269978" (UID: "a829dfbb-7c3e-4c29-8644-87a3d2269978"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:22:12.548036 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.547996 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:12.548036 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.548029 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a829dfbb-7c3e-4c29-8644-87a3d2269978-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:12.548036 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.548040 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:12.548290 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.548049 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a829dfbb-7c3e-4c29-8644-87a3d2269978-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:12.548290 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.548060 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45mcg\" (UniqueName: \"kubernetes.io/projected/a829dfbb-7c3e-4c29-8644-87a3d2269978-kube-api-access-45mcg\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:12.990941 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.990910 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2"] Oct 30 21:22:12.991144 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:12.991119 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" podUID="a2cd625d-00dd-459b-827a-510475c4b444" containerName="istio-proxy" containerID="cri-o://30c5592f723509b1052226ea392a6dbde72828e28da759233f7ad84866015eef" gracePeriod=30 Oct 30 21:22:13.194842 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:13.194737 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" event={"ID":"a829dfbb-7c3e-4c29-8644-87a3d2269978","Type":"ContainerDied","Data":"7676e295e0a069cc288180c0ba05c39d11be40dbf4eef682ae0850a08a4bf02b"} Oct 30 21:22:13.195393 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:13.195372 4848 scope.go:122] "RemoveContainer" containerID="b524d5aeb0703dfd01ef73abc9e9eb6407926dbf65809d01ffbdeae7f92c610c" Oct 30 21:22:13.195674 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:13.195656 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8" Oct 30 21:22:13.226680 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:13.226647 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8"] Oct 30 21:22:13.230519 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:13.230495 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-4t9b8"] Oct 30 21:22:14.004503 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:14.004469 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g"] Oct 30 21:22:14.004739 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:14.004710 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" podUID="0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" containerName="istio-proxy" containerID="cri-o://3374da1831c1b0d991f7b729419dc88a294f7c1a34e01b5d025ad772ff54618a" gracePeriod=30 Oct 30 21:22:14.935556 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:14.935521 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a829dfbb-7c3e-4c29-8644-87a3d2269978" path="/var/lib/kubelet/pods/a829dfbb-7c3e-4c29-8644-87a3d2269978/volumes" Oct 30 21:22:18.310546 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.310464 4848 generic.go:356] "Generic (PLEG): container finished" podID="a2cd625d-00dd-459b-827a-510475c4b444" containerID="30c5592f723509b1052226ea392a6dbde72828e28da759233f7ad84866015eef" exitCode=0 Oct 30 21:22:18.311072 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.310539 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" event={"ID":"a2cd625d-00dd-459b-827a-510475c4b444","Type":"ContainerDied","Data":"30c5592f723509b1052226ea392a6dbde72828e28da759233f7ad84866015eef"} Oct 30 21:22:18.311072 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.310602 4848 scope.go:122] "RemoveContainer" containerID="d118835d7832150d7bbc54a615732c66611aa86c27f1e0298e7a921c10c7cf94" Oct 30 21:22:18.348873 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.348852 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:22:18.500057 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.500027 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2cd625d-00dd-459b-827a-510475c4b444-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2cd625d-00dd-459b-827a-510475c4b444-istio-podinfo\") pod \"a2cd625d-00dd-459b-827a-510475c4b444\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " Oct 30 21:22:18.500234 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.500085 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-credential-socket\") pod \"a2cd625d-00dd-459b-827a-510475c4b444\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " Oct 30 21:22:18.500234 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.500225 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2cd625d-00dd-459b-827a-510475c4b444-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2cd625d-00dd-459b-827a-510475c4b444-istiod-ca-cert\") pod \"a2cd625d-00dd-459b-827a-510475c4b444\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " Oct 30 21:22:18.500357 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.500298 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-workload-certs\") pod \"a2cd625d-00dd-459b-827a-510475c4b444\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " Oct 30 21:22:18.500462 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.500387 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-istio-envoy\") pod \"a2cd625d-00dd-459b-827a-510475c4b444\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " Oct 30 21:22:18.500462 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.500433 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-credential-socket" pod "a2cd625d-00dd-459b-827a-510475c4b444" (UID: "a2cd625d-00dd-459b-827a-510475c4b444"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:22:18.500627 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.500473 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-workload-socket\") pod \"a2cd625d-00dd-459b-827a-510475c4b444\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " Oct 30 21:22:18.500627 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.500526 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2cd625d-00dd-459b-827a-510475c4b444-istio-token\" (UniqueName: \"kubernetes.io/projected/a2cd625d-00dd-459b-827a-510475c4b444-istio-token\") pod \"a2cd625d-00dd-459b-827a-510475c4b444\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " Oct 30 21:22:18.500627 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.500560 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-workload-certs" pod "a2cd625d-00dd-459b-827a-510475c4b444" (UID: "a2cd625d-00dd-459b-827a-510475c4b444"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:22:18.500863 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.500621 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2cd625d-00dd-459b-827a-510475c4b444-istiod-ca-cert" pod "a2cd625d-00dd-459b-827a-510475c4b444" (UID: "a2cd625d-00dd-459b-827a-510475c4b444"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:22:18.500863 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.500639 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2cd625d-00dd-459b-827a-510475c4b444-kube-api-access-hnpxg\" (UniqueName: \"kubernetes.io/projected/a2cd625d-00dd-459b-827a-510475c4b444-kube-api-access-hnpxg\") pod \"a2cd625d-00dd-459b-827a-510475c4b444\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " Oct 30 21:22:18.500863 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.500686 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-istio-data\") pod \"a2cd625d-00dd-459b-827a-510475c4b444\" (UID: \"a2cd625d-00dd-459b-827a-510475c4b444\") " Oct 30 21:22:18.501061 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.501000 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-workload-socket" pod "a2cd625d-00dd-459b-827a-510475c4b444" (UID: "a2cd625d-00dd-459b-827a-510475c4b444"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:22:18.501134 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.501063 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-istio-data" pod "a2cd625d-00dd-459b-827a-510475c4b444" (UID: "a2cd625d-00dd-459b-827a-510475c4b444"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:22:18.501458 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.501436 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:18.501458 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.501457 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:18.501632 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.501467 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:18.501632 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.501478 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2cd625d-00dd-459b-827a-510475c4b444-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:18.501632 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.501487 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:18.502762 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.502734 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2cd625d-00dd-459b-827a-510475c4b444-istio-podinfo" pod "a2cd625d-00dd-459b-827a-510475c4b444" (UID: "a2cd625d-00dd-459b-827a-510475c4b444"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:22:18.502917 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.502899 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2cd625d-00dd-459b-827a-510475c4b444-kube-api-access-hnpxg" pod "a2cd625d-00dd-459b-827a-510475c4b444" (UID: "a2cd625d-00dd-459b-827a-510475c4b444"). InnerVolumeSpecName "kube-api-access-hnpxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:22:18.503205 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.503182 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-istio-envoy" pod "a2cd625d-00dd-459b-827a-510475c4b444" (UID: "a2cd625d-00dd-459b-827a-510475c4b444"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:22:18.503254 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.503187 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2cd625d-00dd-459b-827a-510475c4b444-istio-token" pod "a2cd625d-00dd-459b-827a-510475c4b444" (UID: "a2cd625d-00dd-459b-827a-510475c4b444"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:22:18.602775 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.602692 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2cd625d-00dd-459b-827a-510475c4b444-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:18.602775 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.602721 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnpxg\" (UniqueName: \"kubernetes.io/projected/a2cd625d-00dd-459b-827a-510475c4b444-kube-api-access-hnpxg\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:18.602775 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.602732 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2cd625d-00dd-459b-827a-510475c4b444-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:18.602775 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:18.602742 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2cd625d-00dd-459b-827a-510475c4b444-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:19.298409 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.298386 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 21:22:19.332980 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.332945 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-data\") pod \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " Oct 30 21:22:19.333546 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.333522 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-token\" (UniqueName: \"kubernetes.io/projected/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-token\") pod \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " Oct 30 21:22:19.333733 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.333718 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-workload-socket\") pod \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " Oct 30 21:22:19.333920 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.333903 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-workload-certs\") pod \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " Oct 30 21:22:19.334070 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.334056 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-kube-api-access-zdbmq\" (UniqueName: \"kubernetes.io/projected/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-kube-api-access-zdbmq\") pod \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " Oct 30 21:22:19.334275 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.334258 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-envoy\") pod \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " Oct 30 21:22:19.334393 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.334373 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-podinfo\") pod \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " Oct 30 21:22:19.334511 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.334486 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-credential-socket\") pod \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " Oct 30 21:22:19.334635 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.334558 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istiod-ca-cert\") pod \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\" (UID: \"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b\") " Oct 30 21:22:19.335908 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.335883 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istiod-ca-cert" pod "0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" (UID: "0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:22:19.336319 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.336301 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-data" pod "0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" (UID: "0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:22:19.338649 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.338626 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-workload-socket" pod "0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" (UID: "0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:22:19.338879 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.338861 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-workload-certs" pod "0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" (UID: "0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:22:19.340697 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.340621 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" event={"ID":"a2cd625d-00dd-459b-827a-510475c4b444","Type":"ContainerDied","Data":"9e25b33fa560c786b44b5633e4058bb8328d488faee1fcaa9ecfdad3faa8db67"} Oct 30 21:22:19.340813 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.340796 4848 scope.go:122] "RemoveContainer" containerID="30c5592f723509b1052226ea392a6dbde72828e28da759233f7ad84866015eef" Oct 30 21:22:19.341018 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.341001 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2" Oct 30 21:22:19.341716 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.341685 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-credential-socket" pod "0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" (UID: "0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:22:19.346216 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.346194 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-token" pod "0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" (UID: "0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:22:19.348396 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.348360 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-kube-api-access-zdbmq" pod "0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" (UID: "0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b"). InnerVolumeSpecName "kube-api-access-zdbmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:22:19.348565 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.348546 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-podinfo" pod "0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" (UID: "0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:22:19.348806 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.348742 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-envoy" pod "0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" (UID: "0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:22:19.352143 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.352117 4848 generic.go:356] "Generic (PLEG): container finished" podID="0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" containerID="3374da1831c1b0d991f7b729419dc88a294f7c1a34e01b5d025ad772ff54618a" exitCode=0 Oct 30 21:22:19.352239 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.352185 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" Oct 30 21:22:19.352239 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.352197 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" event={"ID":"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b","Type":"ContainerDied","Data":"3374da1831c1b0d991f7b729419dc88a294f7c1a34e01b5d025ad772ff54618a"} Oct 30 21:22:19.352356 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.352263 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g" event={"ID":"0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b","Type":"ContainerDied","Data":"38f833e4d0f3a1b6954dfc06964978459596beb5372fb0867cf0a4e82e2e5529"} Oct 30 21:22:19.380466 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.380441 4848 scope.go:122] "RemoveContainer" containerID="3374da1831c1b0d991f7b729419dc88a294f7c1a34e01b5d025ad772ff54618a" Oct 30 21:22:19.391810 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.391762 4848 scope.go:122] "RemoveContainer" containerID="841cbf266602699d8a58d63d0a2d6ceea73042b1bf5a9a829ca16cb7c87629dd" Oct 30 21:22:19.392658 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.392634 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2"] Oct 30 21:22:19.400504 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.400479 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-8fa15b-openshift-default-6cf456b647-lj2z2"] Oct 30 21:22:19.404390 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.404365 4848 scope.go:122] "RemoveContainer" containerID="3374da1831c1b0d991f7b729419dc88a294f7c1a34e01b5d025ad772ff54618a" Oct 30 21:22:19.404923 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:22:19.404728 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3374da1831c1b0d991f7b729419dc88a294f7c1a34e01b5d025ad772ff54618a\": container with ID starting with 3374da1831c1b0d991f7b729419dc88a294f7c1a34e01b5d025ad772ff54618a not found: ID does not exist" containerID="3374da1831c1b0d991f7b729419dc88a294f7c1a34e01b5d025ad772ff54618a" Oct 30 21:22:19.404923 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.404777 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3374da1831c1b0d991f7b729419dc88a294f7c1a34e01b5d025ad772ff54618a"} err="failed to get container status \"3374da1831c1b0d991f7b729419dc88a294f7c1a34e01b5d025ad772ff54618a\": rpc error: code = NotFound desc = could not find container \"3374da1831c1b0d991f7b729419dc88a294f7c1a34e01b5d025ad772ff54618a\": container with ID starting with 3374da1831c1b0d991f7b729419dc88a294f7c1a34e01b5d025ad772ff54618a not found: ID does not exist" Oct 30 21:22:19.404923 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.404820 4848 scope.go:122] "RemoveContainer" containerID="841cbf266602699d8a58d63d0a2d6ceea73042b1bf5a9a829ca16cb7c87629dd" Oct 30 21:22:19.405228 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:22:19.405141 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841cbf266602699d8a58d63d0a2d6ceea73042b1bf5a9a829ca16cb7c87629dd\": container with ID starting with 841cbf266602699d8a58d63d0a2d6ceea73042b1bf5a9a829ca16cb7c87629dd not found: ID does not exist" containerID="841cbf266602699d8a58d63d0a2d6ceea73042b1bf5a9a829ca16cb7c87629dd" Oct 30 21:22:19.405228 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.405176 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841cbf266602699d8a58d63d0a2d6ceea73042b1bf5a9a829ca16cb7c87629dd"} err="failed to get container status \"841cbf266602699d8a58d63d0a2d6ceea73042b1bf5a9a829ca16cb7c87629dd\": rpc error: code = NotFound desc = could not find container \"841cbf266602699d8a58d63d0a2d6ceea73042b1bf5a9a829ca16cb7c87629dd\": container with ID starting with 841cbf266602699d8a58d63d0a2d6ceea73042b1bf5a9a829ca16cb7c87629dd not found: ID does not exist" Oct 30 21:22:19.413901 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.413857 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g"] Oct 30 21:22:19.424044 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.424011 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc102e-openshift-default-6bfc9f987f-92x5g"] Oct 30 21:22:19.441161 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.441130 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:19.441161 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.441160 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:19.441403 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.441176 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:19.441403 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.441189 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:19.441403 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.441202 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdbmq\" (UniqueName: \"kubernetes.io/projected/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-kube-api-access-zdbmq\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:19.441403 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.441215 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:19.441403 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.441226 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:19.441403 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.441238 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:19.441403 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:19.441273 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:22:20.936661 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:20.936631 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" path="/var/lib/kubelet/pods/0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b/volumes" Oct 30 21:22:20.937248 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:22:20.937231 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2cd625d-00dd-459b-827a-510475c4b444" path="/var/lib/kubelet/pods/a2cd625d-00dd-459b-827a-510475c4b444/volumes" Oct 30 21:22:22.544761 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:22:22.544716 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:22:52.513348 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:22:52.513306 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:23:16.454832 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:23:16.454799 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 21:23:16.460549 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:23:16.460527 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 21:23:22.597073 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:23:22.597033 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:23:52.514852 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:23:52.514729 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:24:10.376778 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:10.376729 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h"] Oct 30 21:24:10.377430 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:10.377383 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" podUID="78a62f4f-ca51-47f0-8c4a-9d146a810826" containerName="istio-proxy" containerID="cri-o://2b89067dd5537d449efcf2fc389b30fcf11f1935f43635f10869cbfb4c9eab29" gracePeriod=30 Oct 30 21:24:10.384941 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:10.384917 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk"] Oct 30 21:24:10.385242 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:10.385216 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" podUID="c03f9217-ab3a-4aac-a1ea-a04deeb81d36" containerName="istio-proxy" containerID="cri-o://8ade5cd7f2124260dd5f51fa0b58e8039bfacdaea3f3bc7f2195f8b4738e37f1" gracePeriod=30 Oct 30 21:24:14.666335 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:14.666292 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:24:14.666864 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:14.666376 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" podUID="c03f9217-ab3a-4aac-a1ea-a04deeb81d36" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:24:14.759821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:14.759787 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:24:14.760021 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:14.759839 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" podUID="78a62f4f-ca51-47f0-8c4a-9d146a810826" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:24:15.755732 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.755708 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:24:15.763083 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.763053 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:24:15.872967 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.872931 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-token\" (UniqueName: \"kubernetes.io/projected/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-token\") pod \"78a62f4f-ca51-47f0-8c4a-9d146a810826\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " Oct 30 21:24:15.873202 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.872983 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-data\") pod \"78a62f4f-ca51-47f0-8c4a-9d146a810826\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " Oct 30 21:24:15.873202 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.873024 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-workload-certs\") pod \"78a62f4f-ca51-47f0-8c4a-9d146a810826\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " Oct 30 21:24:15.873202 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.873060 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-credential-socket\") pod \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " Oct 30 21:24:15.873202 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.873106 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-data\") pod \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " Oct 30 21:24:15.873202 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.873137 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-workload-certs\") pod \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " Oct 30 21:24:15.873202 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.873188 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-envoy\") pod \"78a62f4f-ca51-47f0-8c4a-9d146a810826\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " Oct 30 21:24:15.873621 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.873259 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-credential-socket\") pod \"78a62f4f-ca51-47f0-8c4a-9d146a810826\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " Oct 30 21:24:15.873621 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.873302 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-token\" (UniqueName: \"kubernetes.io/projected/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-token\") pod \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " Oct 30 21:24:15.873621 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.873335 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-workload-socket\") pod \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " Oct 30 21:24:15.873621 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.873389 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-kube-api-access-k7wng\" (UniqueName: \"kubernetes.io/projected/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-kube-api-access-k7wng\") pod \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " Oct 30 21:24:15.873621 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.873464 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78a62f4f-ca51-47f0-8c4a-9d146a810826-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78a62f4f-ca51-47f0-8c4a-9d146a810826-istiod-ca-cert\") pod \"78a62f4f-ca51-47f0-8c4a-9d146a810826\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " Oct 30 21:24:15.873962 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.873729 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-workload-socket" pod "c03f9217-ab3a-4aac-a1ea-a04deeb81d36" (UID: "c03f9217-ab3a-4aac-a1ea-a04deeb81d36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:15.874055 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.874035 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-workload-certs" pod "c03f9217-ab3a-4aac-a1ea-a04deeb81d36" (UID: "c03f9217-ab3a-4aac-a1ea-a04deeb81d36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:15.874304 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.874281 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-data" pod "c03f9217-ab3a-4aac-a1ea-a04deeb81d36" (UID: "c03f9217-ab3a-4aac-a1ea-a04deeb81d36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:15.874508 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.874407 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a62f4f-ca51-47f0-8c4a-9d146a810826-istiod-ca-cert" pod "78a62f4f-ca51-47f0-8c4a-9d146a810826" (UID: "78a62f4f-ca51-47f0-8c4a-9d146a810826"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:15.874594 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.874546 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-credential-socket" pod "78a62f4f-ca51-47f0-8c4a-9d146a810826" (UID: "78a62f4f-ca51-47f0-8c4a-9d146a810826"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:15.874853 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.874830 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-data" pod "78a62f4f-ca51-47f0-8c4a-9d146a810826" (UID: "78a62f4f-ca51-47f0-8c4a-9d146a810826"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:15.875032 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.875012 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-workload-certs" pod "78a62f4f-ca51-47f0-8c4a-9d146a810826" (UID: "78a62f4f-ca51-47f0-8c4a-9d146a810826"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:15.875475 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.875455 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-credential-socket" pod "c03f9217-ab3a-4aac-a1ea-a04deeb81d36" (UID: "c03f9217-ab3a-4aac-a1ea-a04deeb81d36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:15.875597 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.875481 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-podinfo\") pod \"78a62f4f-ca51-47f0-8c4a-9d146a810826\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " Oct 30 21:24:15.875818 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.875800 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78a62f4f-ca51-47f0-8c4a-9d146a810826-kube-api-access-tpvtx\" (UniqueName: \"kubernetes.io/projected/78a62f4f-ca51-47f0-8c4a-9d146a810826-kube-api-access-tpvtx\") pod \"78a62f4f-ca51-47f0-8c4a-9d146a810826\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " Oct 30 21:24:15.876011 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.875995 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-workload-socket\") pod \"78a62f4f-ca51-47f0-8c4a-9d146a810826\" (UID: \"78a62f4f-ca51-47f0-8c4a-9d146a810826\") " Oct 30 21:24:15.876233 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.876218 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-podinfo\") pod \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " Oct 30 21:24:15.876411 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.876395 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-envoy\") pod \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " Oct 30 21:24:15.876533 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.876517 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istiod-ca-cert\") pod \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\" (UID: \"c03f9217-ab3a-4aac-a1ea-a04deeb81d36\") " Oct 30 21:24:15.877441 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.877412 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-token" pod "c03f9217-ab3a-4aac-a1ea-a04deeb81d36" (UID: "c03f9217-ab3a-4aac-a1ea-a04deeb81d36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:15.877559 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.877481 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:15.877559 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.877507 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:15.877559 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.877524 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:15.877559 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.877539 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:15.877559 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.877553 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:15.877981 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.877580 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:15.877981 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.877593 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:15.877981 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.877608 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78a62f4f-ca51-47f0-8c4a-9d146a810826-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:15.877981 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.877884 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-workload-socket" pod "78a62f4f-ca51-47f0-8c4a-9d146a810826" (UID: "78a62f4f-ca51-47f0-8c4a-9d146a810826"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:15.878187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.878157 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-envoy" pod "78a62f4f-ca51-47f0-8c4a-9d146a810826" (UID: "78a62f4f-ca51-47f0-8c4a-9d146a810826"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:15.878284 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.878185 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istiod-ca-cert" pod "c03f9217-ab3a-4aac-a1ea-a04deeb81d36" (UID: "c03f9217-ab3a-4aac-a1ea-a04deeb81d36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:15.878616 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.878590 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a62f4f-ca51-47f0-8c4a-9d146a810826-kube-api-access-tpvtx" pod "78a62f4f-ca51-47f0-8c4a-9d146a810826" (UID: "78a62f4f-ca51-47f0-8c4a-9d146a810826"). InnerVolumeSpecName "kube-api-access-tpvtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:15.879009 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.878986 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-podinfo" pod "78a62f4f-ca51-47f0-8c4a-9d146a810826" (UID: "78a62f4f-ca51-47f0-8c4a-9d146a810826"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:24:15.879203 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.879179 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-podinfo" pod "c03f9217-ab3a-4aac-a1ea-a04deeb81d36" (UID: "c03f9217-ab3a-4aac-a1ea-a04deeb81d36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:24:15.879416 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.879394 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-kube-api-access-k7wng" pod "c03f9217-ab3a-4aac-a1ea-a04deeb81d36" (UID: "c03f9217-ab3a-4aac-a1ea-a04deeb81d36"). InnerVolumeSpecName "kube-api-access-k7wng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:15.880064 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.880047 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-token" pod "78a62f4f-ca51-47f0-8c4a-9d146a810826" (UID: "78a62f4f-ca51-47f0-8c4a-9d146a810826"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:15.880379 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.880361 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-envoy" pod "c03f9217-ab3a-4aac-a1ea-a04deeb81d36" (UID: "c03f9217-ab3a-4aac-a1ea-a04deeb81d36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:15.891817 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.891747 4848 generic.go:356] "Generic (PLEG): container finished" podID="c03f9217-ab3a-4aac-a1ea-a04deeb81d36" containerID="8ade5cd7f2124260dd5f51fa0b58e8039bfacdaea3f3bc7f2195f8b4738e37f1" exitCode=0 Oct 30 21:24:15.891817 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.891809 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" Oct 30 21:24:15.891936 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.891795 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" event={"ID":"c03f9217-ab3a-4aac-a1ea-a04deeb81d36","Type":"ContainerDied","Data":"8ade5cd7f2124260dd5f51fa0b58e8039bfacdaea3f3bc7f2195f8b4738e37f1"} Oct 30 21:24:15.891986 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.891956 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk" event={"ID":"c03f9217-ab3a-4aac-a1ea-a04deeb81d36","Type":"ContainerDied","Data":"20d87c6f5c39933ac57093f9f0f5c7134168734863080ce047d7ad06a0d985a7"} Oct 30 21:24:15.892023 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.891983 4848 scope.go:122] "RemoveContainer" containerID="8ade5cd7f2124260dd5f51fa0b58e8039bfacdaea3f3bc7f2195f8b4738e37f1" Oct 30 21:24:15.893709 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.893686 4848 generic.go:356] "Generic (PLEG): container finished" podID="78a62f4f-ca51-47f0-8c4a-9d146a810826" containerID="2b89067dd5537d449efcf2fc389b30fcf11f1935f43635f10869cbfb4c9eab29" exitCode=0 Oct 30 21:24:15.893810 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.893725 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" Oct 30 21:24:15.893810 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.893732 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" event={"ID":"78a62f4f-ca51-47f0-8c4a-9d146a810826","Type":"ContainerDied","Data":"2b89067dd5537d449efcf2fc389b30fcf11f1935f43635f10869cbfb4c9eab29"} Oct 30 21:24:15.893940 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.893853 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h" event={"ID":"78a62f4f-ca51-47f0-8c4a-9d146a810826","Type":"ContainerDied","Data":"c136f43b861b1426c9eaccbc0b55f308da62ef43cab3e877ebfc61327a1ffc8f"} Oct 30 21:24:15.904089 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.904063 4848 scope.go:122] "RemoveContainer" containerID="8ade5cd7f2124260dd5f51fa0b58e8039bfacdaea3f3bc7f2195f8b4738e37f1" Oct 30 21:24:15.904387 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:24:15.904369 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ade5cd7f2124260dd5f51fa0b58e8039bfacdaea3f3bc7f2195f8b4738e37f1\": container with ID starting with 8ade5cd7f2124260dd5f51fa0b58e8039bfacdaea3f3bc7f2195f8b4738e37f1 not found: ID does not exist" containerID="8ade5cd7f2124260dd5f51fa0b58e8039bfacdaea3f3bc7f2195f8b4738e37f1" Oct 30 21:24:15.904456 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.904395 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ade5cd7f2124260dd5f51fa0b58e8039bfacdaea3f3bc7f2195f8b4738e37f1"} err="failed to get container status \"8ade5cd7f2124260dd5f51fa0b58e8039bfacdaea3f3bc7f2195f8b4738e37f1\": rpc error: code = NotFound desc = could not find container \"8ade5cd7f2124260dd5f51fa0b58e8039bfacdaea3f3bc7f2195f8b4738e37f1\": container with ID starting with 8ade5cd7f2124260dd5f51fa0b58e8039bfacdaea3f3bc7f2195f8b4738e37f1 not found: ID does not exist" Oct 30 21:24:15.904456 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.904412 4848 scope.go:122] "RemoveContainer" containerID="2b89067dd5537d449efcf2fc389b30fcf11f1935f43635f10869cbfb4c9eab29" Oct 30 21:24:15.914483 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.914466 4848 scope.go:122] "RemoveContainer" containerID="2b89067dd5537d449efcf2fc389b30fcf11f1935f43635f10869cbfb4c9eab29" Oct 30 21:24:15.914737 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:24:15.914720 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b89067dd5537d449efcf2fc389b30fcf11f1935f43635f10869cbfb4c9eab29\": container with ID starting with 2b89067dd5537d449efcf2fc389b30fcf11f1935f43635f10869cbfb4c9eab29 not found: ID does not exist" containerID="2b89067dd5537d449efcf2fc389b30fcf11f1935f43635f10869cbfb4c9eab29" Oct 30 21:24:15.914817 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.914748 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b89067dd5537d449efcf2fc389b30fcf11f1935f43635f10869cbfb4c9eab29"} err="failed to get container status \"2b89067dd5537d449efcf2fc389b30fcf11f1935f43635f10869cbfb4c9eab29\": rpc error: code = NotFound desc = could not find container \"2b89067dd5537d449efcf2fc389b30fcf11f1935f43635f10869cbfb4c9eab29\": container with ID starting with 2b89067dd5537d449efcf2fc389b30fcf11f1935f43635f10869cbfb4c9eab29 not found: ID does not exist" Oct 30 21:24:15.977459 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.977396 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h"] Oct 30 21:24:15.979079 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.979060 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:15.979130 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.979083 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:15.979130 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.979093 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:15.979130 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.979101 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:15.979130 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.979111 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:15.979130 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.979118 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:15.979130 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.979129 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:15.979323 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.979137 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7wng\" (UniqueName: \"kubernetes.io/projected/c03f9217-ab3a-4aac-a1ea-a04deeb81d36-kube-api-access-k7wng\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:15.979323 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.979147 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78a62f4f-ca51-47f0-8c4a-9d146a810826-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:15.979323 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:15.979156 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpvtx\" (UniqueName: \"kubernetes.io/projected/78a62f4f-ca51-47f0-8c4a-9d146a810826-kube-api-access-tpvtx\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:16.008567 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:16.008538 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-j8z9h"] Oct 30 21:24:16.049671 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:16.049642 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk"] Oct 30 21:24:16.053632 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:16.053610 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-345bc4-openshift-default-c59dc5c89-wdjgk"] Oct 30 21:24:16.972752 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:16.972660 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78a62f4f-ca51-47f0-8c4a-9d146a810826" path="/var/lib/kubelet/pods/78a62f4f-ca51-47f0-8c4a-9d146a810826/volumes" Oct 30 21:24:16.974234 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:16.974192 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c03f9217-ab3a-4aac-a1ea-a04deeb81d36" path="/var/lib/kubelet/pods/c03f9217-ab3a-4aac-a1ea-a04deeb81d36/volumes" Oct 30 21:24:20.336791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:20.330866 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-fmzmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:24:20.336791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:20.330936 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-fmzmj" podUID="7dc1d3fc-a66d-41d9-b81c-d17801d8702e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:20.692940 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:20.692902 4848 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-sb5c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:24:20.693157 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:20.692976 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-sb5c9" podUID="a6fc6562-45cb-47db-98ab-415e184c71b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:21.034804 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:21.034707 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg"] Oct 30 21:24:22.519197 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:24:22.519162 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:24:26.035338 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:26.035282 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg" podUID="a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6" containerName="payload-processing" containerID="cri-o://ac0c55433391c09afe73ca33f5344ba831cbced5cde43f74d475d0e65ade201b" gracePeriod=25 Oct 30 21:24:28.991724 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:28.991685 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg" podUID="a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:24:36.264388 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.264354 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw"] Oct 30 21:24:36.264895 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.264607 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" podUID="83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18" containerName="maas-api" containerID="cri-o://e2aa53b437beb9b90331b5aec4e696f56455d98dce7f9c1078909173d0f9c8b2" gracePeriod=30 Oct 30 21:24:36.833469 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.832833 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf"] Oct 30 21:24:36.833929 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.833908 4848 cpu_manager.go:414] "containerMap: removing container" podUID="a829dfbb-7c3e-4c29-8644-87a3d2269978" containerName="istio-proxy" Oct 30 21:24:36.834096 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.834082 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a829dfbb-7c3e-4c29-8644-87a3d2269978" containerName="istio-proxy" Oct 30 21:24:36.834221 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.834210 4848 cpu_manager.go:414] "containerMap: removing container" podUID="78a62f4f-ca51-47f0-8c4a-9d146a810826" containerName="istio-proxy" Oct 30 21:24:36.834297 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.834288 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78a62f4f-ca51-47f0-8c4a-9d146a810826" containerName="istio-proxy" Oct 30 21:24:36.834402 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.834393 4848 cpu_manager.go:414] "containerMap: removing container" podUID="a2cd625d-00dd-459b-827a-510475c4b444" containerName="istio-proxy" Oct 30 21:24:36.834746 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.834730 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2cd625d-00dd-459b-827a-510475c4b444" containerName="istio-proxy" Oct 30 21:24:36.834902 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.834891 4848 cpu_manager.go:414] "containerMap: removing container" podUID="0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" containerName="istio-proxy" Oct 30 21:24:36.834984 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.834974 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" containerName="istio-proxy" Oct 30 21:24:36.835063 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.835055 4848 cpu_manager.go:414] "containerMap: removing container" podUID="a2cd625d-00dd-459b-827a-510475c4b444" containerName="istio-proxy" Oct 30 21:24:36.835136 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.835125 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2cd625d-00dd-459b-827a-510475c4b444" containerName="istio-proxy" Oct 30 21:24:36.835217 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.835208 4848 cpu_manager.go:414] "containerMap: removing container" podUID="0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" containerName="istio-proxy" Oct 30 21:24:36.835288 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.835279 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" containerName="istio-proxy" Oct 30 21:24:36.835409 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.835399 4848 cpu_manager.go:414] "containerMap: removing container" podUID="c03f9217-ab3a-4aac-a1ea-a04deeb81d36" containerName="istio-proxy" Oct 30 21:24:36.835494 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.835484 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c03f9217-ab3a-4aac-a1ea-a04deeb81d36" containerName="istio-proxy" Oct 30 21:24:36.835780 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.835755 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="a829dfbb-7c3e-4c29-8644-87a3d2269978" containerName="istio-proxy" Oct 30 21:24:36.835889 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.835879 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="78a62f4f-ca51-47f0-8c4a-9d146a810826" containerName="istio-proxy" Oct 30 21:24:36.835973 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.835965 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2cd625d-00dd-459b-827a-510475c4b444" containerName="istio-proxy" Oct 30 21:24:36.836051 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.836043 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" containerName="istio-proxy" Oct 30 21:24:36.836125 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.836117 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2cd625d-00dd-459b-827a-510475c4b444" containerName="istio-proxy" Oct 30 21:24:36.836201 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.836193 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bdf57f3-fcef-4f4c-82b3-3b8ce7e3882b" containerName="istio-proxy" Oct 30 21:24:36.836278 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.836270 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="c03f9217-ab3a-4aac-a1ea-a04deeb81d36" containerName="istio-proxy" Oct 30 21:24:36.840673 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.840608 4848 generic.go:356] "Generic (PLEG): container finished" podID="83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18" containerID="e2aa53b437beb9b90331b5aec4e696f56455d98dce7f9c1078909173d0f9c8b2" exitCode=0 Oct 30 21:24:36.842173 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.842149 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" event={"ID":"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18","Type":"ContainerDied","Data":"e2aa53b437beb9b90331b5aec4e696f56455d98dce7f9c1078909173d0f9c8b2"} Oct 30 21:24:36.842407 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.842393 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" Oct 30 21:24:36.847239 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.847216 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5506f4-serving-cert\"" Oct 30 21:24:36.855222 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.855198 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf"] Oct 30 21:24:36.860912 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.860894 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" Oct 30 21:24:36.892227 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.892188 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd7a438e-6705-4f55-8580-fafb7a21f4f0-maas-api-tls\") pod \"maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf\" (UID: \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" Oct 30 21:24:36.892406 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.892379 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcjbb\" (UniqueName: \"kubernetes.io/projected/fd7a438e-6705-4f55-8580-fafb7a21f4f0-kube-api-access-qcjbb\") pod \"maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf\" (UID: \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" Oct 30 21:24:36.892625 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.892606 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd7a438e-6705-4f55-8580-fafb7a21f4f0-metrics-certs\") pod \"maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf\" (UID: \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" Oct 30 21:24:36.892823 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.892781 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd7a438e-6705-4f55-8580-fafb7a21f4f0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf\" (UID: \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" Oct 30 21:24:36.994195 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.994108 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-kube-api-access-7g29n\" (UniqueName: \"kubernetes.io/projected/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-kube-api-access-7g29n\") pod \"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18\" (UID: \"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18\") " Oct 30 21:24:36.994382 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.994301 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-maas-api-tls\") pod \"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18\" (UID: \"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18\") " Oct 30 21:24:36.994581 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.994548 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-metrics-certs\" (UniqueName: \"kubernetes.io/secret/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-metrics-certs\") pod \"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18\" (UID: \"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18\") " Oct 30 21:24:36.995239 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.995191 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-openshift-service-ca\") pod \"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18\" (UID: \"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18\") " Oct 30 21:24:36.995606 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.995571 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-openshift-service-ca" pod "83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18" (UID: "83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:36.995714 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.995662 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd7a438e-6705-4f55-8580-fafb7a21f4f0-metrics-certs\") pod \"maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf\" (UID: \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" Oct 30 21:24:36.997168 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.995902 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd7a438e-6705-4f55-8580-fafb7a21f4f0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf\" (UID: \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" Oct 30 21:24:36.997168 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.995984 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd7a438e-6705-4f55-8580-fafb7a21f4f0-maas-api-tls\") pod \"maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf\" (UID: \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" Oct 30 21:24:36.997168 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.996208 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcjbb\" (UniqueName: \"kubernetes.io/projected/fd7a438e-6705-4f55-8580-fafb7a21f4f0-kube-api-access-qcjbb\") pod \"maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf\" (UID: \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" Oct 30 21:24:36.997168 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.996358 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:36.997168 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:24:36.996882 4848 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-serving-cert: secret "maas-api-e2e-inf-b-5506f4-serving-cert" not found Oct 30 21:24:36.997168 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:24:36.996948 4848 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd7a438e-6705-4f55-8580-fafb7a21f4f0-maas-api-tls podName:fd7a438e-6705-4f55-8580-fafb7a21f4f0 nodeName:}" failed. No retries permitted until 2026-10-30 21:24:37.496930453 +0000 UTC m=+2782.676826951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd7a438e-6705-4f55-8580-fafb7a21f4f0-maas-api-tls") pod "maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" (UID: "fd7a438e-6705-4f55-8580-fafb7a21f4f0") : secret "maas-api-e2e-inf-b-5506f4-serving-cert" not found Oct 30 21:24:36.997168 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.996972 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd7a438e-6705-4f55-8580-fafb7a21f4f0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf\" (UID: \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" Oct 30 21:24:36.997568 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.997407 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-metrics-certs" pod "83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18" (UID: "83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:36.997568 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.997444 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-maas-api-tls" pod "83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18" (UID: "83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:36.997568 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.997486 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-kube-api-access-7g29n" pod "83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18" (UID: "83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18"). InnerVolumeSpecName "kube-api-access-7g29n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:36.999376 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:36.999357 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd7a438e-6705-4f55-8580-fafb7a21f4f0-metrics-certs\") pod \"maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf\" (UID: \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" Oct 30 21:24:37.062789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:37.047680 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcjbb\" (UniqueName: \"kubernetes.io/projected/fd7a438e-6705-4f55-8580-fafb7a21f4f0-kube-api-access-qcjbb\") pod \"maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf\" (UID: \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" Oct 30 21:24:37.097808 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:37.097756 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:37.097808 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:37.097809 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:37.098048 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:37.097825 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g29n\" (UniqueName: \"kubernetes.io/projected/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18-kube-api-access-7g29n\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:37.504213 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:37.504176 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd7a438e-6705-4f55-8580-fafb7a21f4f0-maas-api-tls\") pod \"maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf\" (UID: \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" Oct 30 21:24:37.507031 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:37.507005 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd7a438e-6705-4f55-8580-fafb7a21f4f0-maas-api-tls\") pod \"maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf\" (UID: \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" Oct 30 21:24:37.781002 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:37.780964 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" Oct 30 21:24:37.883278 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:37.883249 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" event={"ID":"83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18","Type":"ContainerDied","Data":"819a7f9e48f70aac41be9374002c199b37324df3eae37846b9168c86cd6d2ac3"} Oct 30 21:24:37.883518 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:37.883506 4848 scope.go:122] "RemoveContainer" containerID="e2aa53b437beb9b90331b5aec4e696f56455d98dce7f9c1078909173d0f9c8b2" Oct 30 21:24:37.883665 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:37.883295 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw" Oct 30 21:24:37.923229 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:37.918204 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw"] Oct 30 21:24:37.930578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:37.930459 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a574c2-65b457cb5-trpzw"] Oct 30 21:24:38.152031 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:38.152006 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf"] Oct 30 21:24:38.155313 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:24:38.155289 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7a438e_6705_4f55_8580_fafb7a21f4f0.slice/crio-e0e8bcc35e4e04e770bcf8e8164fe378a2d819f151d5796d53d355638b7b7c64 WatchSource:0}: Error finding container e0e8bcc35e4e04e770bcf8e8164fe378a2d819f151d5796d53d355638b7b7c64: Status 404 returned error can't find the container with id e0e8bcc35e4e04e770bcf8e8164fe378a2d819f151d5796d53d355638b7b7c64 Oct 30 21:24:38.156728 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:38.156710 4848 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:24:38.949275 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:38.949241 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18" path="/var/lib/kubelet/pods/83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18/volumes" Oct 30 21:24:38.949703 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:38.949685 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" Oct 30 21:24:38.949758 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:38.949742 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" event={"ID":"fd7a438e-6705-4f55-8580-fafb7a21f4f0","Type":"ContainerStarted","Data":"a5e567571038c54ede10760341e7a856808dd38f65ad95616a7adc1850ebcdfd"} Oct 30 21:24:38.949842 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:38.949789 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" event={"ID":"fd7a438e-6705-4f55-8580-fafb7a21f4f0","Type":"ContainerStarted","Data":"e0e8bcc35e4e04e770bcf8e8164fe378a2d819f151d5796d53d355638b7b7c64"} Oct 30 21:24:38.955001 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:38.954956 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" podStartSLOduration=2.569284431 podStartE2EDuration="2.954942707s" totalImagesPullingTime="385.658276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:38.95198443 +0000 UTC m=+2784.131880939" watchObservedRunningTime="2026-10-30 21:24:38.954942707 +0000 UTC m=+2784.134839214" Oct 30 21:24:38.991115 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:38.991075 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg" podUID="a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:24:44.194610 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:44.194571 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x"] Oct 30 21:24:44.195149 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:44.194802 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" podUID="05d85436-702d-4fb3-8e98-220cb26ef773" containerName="istio-proxy" containerID="cri-o://3bbee5d997338eea519f63b8e9493cecc1b6a085cecbc61b37179ff81acb5103" gracePeriod=30 Oct 30 21:24:44.946231 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:44.946202 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" Oct 30 21:24:45.758437 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.758405 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf"] Oct 30 21:24:45.760898 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.760872 4848 cpu_manager.go:414] "containerMap: removing container" podUID="83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18" containerName="maas-api" Oct 30 21:24:45.762812 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.762790 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18" containerName="maas-api" Oct 30 21:24:45.763404 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.763339 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="83cf9cb4-12c1-47dc-8e95-d1ca0ee6ea18" containerName="maas-api" Oct 30 21:24:45.794065 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.794017 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf"] Oct 30 21:24:45.794227 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.794140 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:45.812190 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.812149 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a3fb70-openshift-default-dockercfg-clgk4\"" Oct 30 21:24:45.888513 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.888477 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-podinfo\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:45.888702 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.888527 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-workload-certs\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:45.888702 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.888660 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjdt6\" (UniqueName: \"kubernetes.io/projected/b02fca84-f078-4d46-a842-a9b29f63eaaa-kube-api-access-bjdt6\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:45.888831 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.888721 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-credential-socket\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:45.888893 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.888870 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-envoy\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:45.889031 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.889010 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b02fca84-f078-4d46-a842-a9b29f63eaaa-istiod-ca-cert\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:45.889090 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.889051 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-workload-socket\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:45.889137 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.889085 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-data\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:45.889176 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.889138 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-token\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:45.996789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.996428 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-podinfo\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:45.996789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.996482 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-workload-certs\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:45.996789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.996553 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjdt6\" (UniqueName: \"kubernetes.io/projected/b02fca84-f078-4d46-a842-a9b29f63eaaa-kube-api-access-bjdt6\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:45.996789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.996612 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-credential-socket\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:45.996789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.996640 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-envoy\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:45.996789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.996785 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b02fca84-f078-4d46-a842-a9b29f63eaaa-istiod-ca-cert\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:45.997273 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.996827 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-workload-socket\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:45.997273 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.996871 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-data\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:45.997273 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:45.996927 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-token\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:46.012843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.012747 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-workload-certs\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:46.031811 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.014733 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-credential-socket\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:46.031811 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.014808 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-data\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:46.031811 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.015348 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b02fca84-f078-4d46-a842-a9b29f63eaaa-istiod-ca-cert\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:46.031811 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.017457 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-podinfo\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:46.032440 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.032408 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-workload-socket\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:46.055591 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.055561 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjdt6\" (UniqueName: \"kubernetes.io/projected/b02fca84-f078-4d46-a842-a9b29f63eaaa-kube-api-access-bjdt6\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:46.077272 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.077238 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-envoy\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:46.077540 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.077513 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-token\") pod \"e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:46.137294 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.136081 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:46.486534 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.486500 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg" Oct 30 21:24:46.529580 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.529484 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf"] Oct 30 21:24:46.531046 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:24:46.531015 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02fca84_f078_4d46_a842_a9b29f63eaaa.slice/crio-34eb69b8d466c7f2fc36ab7e374d88e9f10e5c0b8f2a77d487265c95980bc3a0 WatchSource:0}: Error finding container 34eb69b8d466c7f2fc36ab7e374d88e9f10e5c0b8f2a77d487265c95980bc3a0: Status 404 returned error can't find the container with id 34eb69b8d466c7f2fc36ab7e374d88e9f10e5c0b8f2a77d487265c95980bc3a0 Oct 30 21:24:46.534901 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.534861 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:24:46.535023 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.534934 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:24:46.535023 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.534964 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:24:46.630110 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.630076 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6-kube-api-access-tbqcx\" (UniqueName: \"kubernetes.io/projected/a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6-kube-api-access-tbqcx\") pod \"a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6\" (UID: \"a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6\") " Oct 30 21:24:46.630328 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.630305 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6-plugins-config-volume\") pod \"a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6\" (UID: \"a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6\") " Oct 30 21:24:46.630866 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.630841 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6-plugins-config-volume" pod "a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6" (UID: "a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:46.631113 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.631092 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:46.632938 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.632890 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6-kube-api-access-tbqcx" pod "a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6" (UID: "a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6"). InnerVolumeSpecName "kube-api-access-tbqcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:46.732715 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:46.732679 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbqcx\" (UniqueName: \"kubernetes.io/projected/a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6-kube-api-access-tbqcx\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:47.108736 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.108698 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" event={"ID":"b02fca84-f078-4d46-a842-a9b29f63eaaa","Type":"ContainerStarted","Data":"0a84b750ea01dadef8d7b57baf99397ec6cfe24d43c7e17c771770418df3b565"} Oct 30 21:24:47.109239 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.108759 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" event={"ID":"b02fca84-f078-4d46-a842-a9b29f63eaaa","Type":"ContainerStarted","Data":"34eb69b8d466c7f2fc36ab7e374d88e9f10e5c0b8f2a77d487265c95980bc3a0"} Oct 30 21:24:47.110317 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.110288 4848 generic.go:356] "Generic (PLEG): container finished" podID="a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6" containerID="ac0c55433391c09afe73ca33f5344ba831cbced5cde43f74d475d0e65ade201b" exitCode=0 Oct 30 21:24:47.110425 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.110326 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg" Oct 30 21:24:47.110425 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.110339 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg" event={"ID":"a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6","Type":"ContainerDied","Data":"ac0c55433391c09afe73ca33f5344ba831cbced5cde43f74d475d0e65ade201b"} Oct 30 21:24:47.110425 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.110387 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg" event={"ID":"a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6","Type":"ContainerDied","Data":"b4533d59c34a8b601ab2696a5bba6711ca5bdda07f95dda946e741e4b0cb8bd1"} Oct 30 21:24:47.110425 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.110417 4848 scope.go:122] "RemoveContainer" containerID="ac0c55433391c09afe73ca33f5344ba831cbced5cde43f74d475d0e65ade201b" Oct 30 21:24:47.127564 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.127541 4848 scope.go:122] "RemoveContainer" containerID="ac0c55433391c09afe73ca33f5344ba831cbced5cde43f74d475d0e65ade201b" Oct 30 21:24:47.127991 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:24:47.127959 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac0c55433391c09afe73ca33f5344ba831cbced5cde43f74d475d0e65ade201b\": container with ID starting with ac0c55433391c09afe73ca33f5344ba831cbced5cde43f74d475d0e65ade201b not found: ID does not exist" containerID="ac0c55433391c09afe73ca33f5344ba831cbced5cde43f74d475d0e65ade201b" Oct 30 21:24:47.128083 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.128001 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac0c55433391c09afe73ca33f5344ba831cbced5cde43f74d475d0e65ade201b"} err="failed to get container status \"ac0c55433391c09afe73ca33f5344ba831cbced5cde43f74d475d0e65ade201b\": rpc error: code = NotFound desc = could not find container \"ac0c55433391c09afe73ca33f5344ba831cbced5cde43f74d475d0e65ade201b\": container with ID starting with ac0c55433391c09afe73ca33f5344ba831cbced5cde43f74d475d0e65ade201b not found: ID does not exist" Oct 30 21:24:47.132089 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.132044 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" podStartSLOduration=2.132032024 podStartE2EDuration="2.132032024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:47.129085479 +0000 UTC m=+2792.308981987" watchObservedRunningTime="2026-10-30 21:24:47.132032024 +0000 UTC m=+2792.311928597" Oct 30 21:24:47.138948 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.138916 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:47.149888 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.149862 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg"] Oct 30 21:24:47.150084 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.150064 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:47.154291 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.154270 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a574c2-5b8867994f-rbwqg"] Oct 30 21:24:47.209078 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.209050 4848 patch_prober.go:29] interesting pod/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:24:47.209242 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.209108 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" podUID="05d85436-702d-4fb3-8e98-220cb26ef773" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:24:47.588653 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.588606 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck"] Oct 30 21:24:47.589869 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.589842 4848 cpu_manager.go:414] "containerMap: removing container" podUID="a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6" containerName="payload-processing" Oct 30 21:24:47.589869 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.589867 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6" containerName="payload-processing" Oct 30 21:24:47.590105 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.590090 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6" containerName="payload-processing" Oct 30 21:24:47.604042 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.604015 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck"] Oct 30 21:24:47.604231 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.604131 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" Oct 30 21:24:47.609816 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.609252 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9b78e4-serving-cert\"" Oct 30 21:24:47.803085 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.803041 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-maas-api-tls\") pod \"maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck\" (UID: \"906f25cf-0aea-484b-9dcc-5302dfb1c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" Oct 30 21:24:47.804343 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.804286 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9f4z\" (UniqueName: \"kubernetes.io/projected/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-kube-api-access-v9f4z\") pod \"maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck\" (UID: \"906f25cf-0aea-484b-9dcc-5302dfb1c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" Oct 30 21:24:47.804704 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.804684 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-metrics-certs\") pod \"maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck\" (UID: \"906f25cf-0aea-484b-9dcc-5302dfb1c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" Oct 30 21:24:47.805151 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.805129 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck\" (UID: \"906f25cf-0aea-484b-9dcc-5302dfb1c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" Oct 30 21:24:47.906822 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.906708 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck\" (UID: \"906f25cf-0aea-484b-9dcc-5302dfb1c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" Oct 30 21:24:47.906992 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.906874 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-maas-api-tls\") pod \"maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck\" (UID: \"906f25cf-0aea-484b-9dcc-5302dfb1c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" Oct 30 21:24:47.906992 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.906904 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9f4z\" (UniqueName: \"kubernetes.io/projected/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-kube-api-access-v9f4z\") pod \"maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck\" (UID: \"906f25cf-0aea-484b-9dcc-5302dfb1c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" Oct 30 21:24:47.906992 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.906986 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-metrics-certs\") pod \"maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck\" (UID: \"906f25cf-0aea-484b-9dcc-5302dfb1c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" Oct 30 21:24:47.907390 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.907365 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck\" (UID: \"906f25cf-0aea-484b-9dcc-5302dfb1c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" Oct 30 21:24:47.909441 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.909415 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-metrics-certs\") pod \"maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck\" (UID: \"906f25cf-0aea-484b-9dcc-5302dfb1c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" Oct 30 21:24:47.909547 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.909528 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-maas-api-tls\") pod \"maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck\" (UID: \"906f25cf-0aea-484b-9dcc-5302dfb1c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" Oct 30 21:24:47.916308 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.916287 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9f4z\" (UniqueName: \"kubernetes.io/projected/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-kube-api-access-v9f4z\") pod \"maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck\" (UID: \"906f25cf-0aea-484b-9dcc-5302dfb1c5e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" Oct 30 21:24:47.922273 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:47.922252 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" Oct 30 21:24:48.119832 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:48.119799 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:48.120921 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:48.120897 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:24:48.216813 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:48.214885 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck"] Oct 30 21:24:48.216813 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:24:48.215314 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906f25cf_0aea_484b_9dcc_5302dfb1c5e4.slice/crio-3016be2d2959606034866009f6b1f2edc2596da6ebcd88faf4cb81c8619cca42 WatchSource:0}: Error finding container 3016be2d2959606034866009f6b1f2edc2596da6ebcd88faf4cb81c8619cca42: Status 404 returned error can't find the container with id 3016be2d2959606034866009f6b1f2edc2596da6ebcd88faf4cb81c8619cca42 Oct 30 21:24:48.934992 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:48.934954 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6" path="/var/lib/kubelet/pods/a29d6d7f-34e8-4bfa-aabe-15e84c94c8f6/volumes" Oct 30 21:24:49.129951 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.129869 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" event={"ID":"906f25cf-0aea-484b-9dcc-5302dfb1c5e4","Type":"ContainerStarted","Data":"6d7ffcdd1bd03b3d28383a05398c9cb16d492000b625595c4374ced043b1a899"} Oct 30 21:24:49.129951 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.129931 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" event={"ID":"906f25cf-0aea-484b-9dcc-5302dfb1c5e4","Type":"ContainerStarted","Data":"3016be2d2959606034866009f6b1f2edc2596da6ebcd88faf4cb81c8619cca42"} Oct 30 21:24:49.152939 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.152892 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" podStartSLOduration=1.564897676 podStartE2EDuration="2.152878411s" totalImagesPullingTime="587.980735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:49.150206389 +0000 UTC m=+2794.330102896" watchObservedRunningTime="2026-10-30 21:24:49.152878411 +0000 UTC m=+2794.332774967" Oct 30 21:24:49.534961 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.534438 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:24:49.644099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.644020 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-credential-socket\") pod \"05d85436-702d-4fb3-8e98-220cb26ef773\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " Oct 30 21:24:49.644250 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.644126 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d85436-702d-4fb3-8e98-220cb26ef773-istio-token\" (UniqueName: \"kubernetes.io/projected/05d85436-702d-4fb3-8e98-220cb26ef773-istio-token\") pod \"05d85436-702d-4fb3-8e98-220cb26ef773\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " Oct 30 21:24:49.644250 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.644187 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-istio-envoy\") pod \"05d85436-702d-4fb3-8e98-220cb26ef773\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " Oct 30 21:24:49.644395 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.644254 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-workload-certs\") pod \"05d85436-702d-4fb3-8e98-220cb26ef773\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " Oct 30 21:24:49.644395 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.644293 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-workload-socket\") pod \"05d85436-702d-4fb3-8e98-220cb26ef773\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " Oct 30 21:24:49.644395 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.644331 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-credential-socket" pod "05d85436-702d-4fb3-8e98-220cb26ef773" (UID: "05d85436-702d-4fb3-8e98-220cb26ef773"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:49.644395 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.644338 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05d85436-702d-4fb3-8e98-220cb26ef773-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d85436-702d-4fb3-8e98-220cb26ef773-istiod-ca-cert\") pod \"05d85436-702d-4fb3-8e98-220cb26ef773\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " Oct 30 21:24:49.644609 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.644469 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d85436-702d-4fb3-8e98-220cb26ef773-kube-api-access-8vwr4\" (UniqueName: \"kubernetes.io/projected/05d85436-702d-4fb3-8e98-220cb26ef773-kube-api-access-8vwr4\") pod \"05d85436-702d-4fb3-8e98-220cb26ef773\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " Oct 30 21:24:49.644609 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.644535 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05d85436-702d-4fb3-8e98-220cb26ef773-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d85436-702d-4fb3-8e98-220cb26ef773-istio-podinfo\") pod \"05d85436-702d-4fb3-8e98-220cb26ef773\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " Oct 30 21:24:49.644719 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.644627 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-istio-data\") pod \"05d85436-702d-4fb3-8e98-220cb26ef773\" (UID: \"05d85436-702d-4fb3-8e98-220cb26ef773\") " Oct 30 21:24:49.644719 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.644654 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-workload-socket" pod "05d85436-702d-4fb3-8e98-220cb26ef773" (UID: "05d85436-702d-4fb3-8e98-220cb26ef773"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:49.644974 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.644949 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-istio-data" pod "05d85436-702d-4fb3-8e98-220cb26ef773" (UID: "05d85436-702d-4fb3-8e98-220cb26ef773"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:49.645075 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.644969 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-workload-certs" pod "05d85436-702d-4fb3-8e98-220cb26ef773" (UID: "05d85436-702d-4fb3-8e98-220cb26ef773"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:49.645246 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.645218 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d85436-702d-4fb3-8e98-220cb26ef773-istiod-ca-cert" pod "05d85436-702d-4fb3-8e98-220cb26ef773" (UID: "05d85436-702d-4fb3-8e98-220cb26ef773"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:49.645482 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.645467 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:49.645562 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.645484 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d85436-702d-4fb3-8e98-220cb26ef773-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:49.645562 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.645493 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:49.645562 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.645503 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:49.645562 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.645511 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:49.646719 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.646689 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-istio-envoy" pod "05d85436-702d-4fb3-8e98-220cb26ef773" (UID: "05d85436-702d-4fb3-8e98-220cb26ef773"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:49.646839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.646804 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05d85436-702d-4fb3-8e98-220cb26ef773-istio-podinfo" pod "05d85436-702d-4fb3-8e98-220cb26ef773" (UID: "05d85436-702d-4fb3-8e98-220cb26ef773"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:24:49.647081 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.647059 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d85436-702d-4fb3-8e98-220cb26ef773-istio-token" pod "05d85436-702d-4fb3-8e98-220cb26ef773" (UID: "05d85436-702d-4fb3-8e98-220cb26ef773"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:49.647175 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.647088 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d85436-702d-4fb3-8e98-220cb26ef773-kube-api-access-8vwr4" pod "05d85436-702d-4fb3-8e98-220cb26ef773" (UID: "05d85436-702d-4fb3-8e98-220cb26ef773"). InnerVolumeSpecName "kube-api-access-8vwr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:49.747298 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.747261 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vwr4\" (UniqueName: \"kubernetes.io/projected/05d85436-702d-4fb3-8e98-220cb26ef773-kube-api-access-8vwr4\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:49.747298 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.747293 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d85436-702d-4fb3-8e98-220cb26ef773-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:49.747298 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.747305 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d85436-702d-4fb3-8e98-220cb26ef773-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:49.747537 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:49.747314 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d85436-702d-4fb3-8e98-220cb26ef773-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:24:50.139341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:50.139308 4848 generic.go:356] "Generic (PLEG): container finished" podID="05d85436-702d-4fb3-8e98-220cb26ef773" containerID="3bbee5d997338eea519f63b8e9493cecc1b6a085cecbc61b37179ff81acb5103" exitCode=0 Oct 30 21:24:50.139737 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:50.139359 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" Oct 30 21:24:50.139737 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:50.139396 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" event={"ID":"05d85436-702d-4fb3-8e98-220cb26ef773","Type":"ContainerDied","Data":"3bbee5d997338eea519f63b8e9493cecc1b6a085cecbc61b37179ff81acb5103"} Oct 30 21:24:50.139737 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:50.139455 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x" event={"ID":"05d85436-702d-4fb3-8e98-220cb26ef773","Type":"ContainerDied","Data":"3987d5eda915e483d5acb4b75a81edb677219a6ba54bbf535551653f2d78b856"} Oct 30 21:24:50.139737 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:50.139484 4848 scope.go:122] "RemoveContainer" containerID="3bbee5d997338eea519f63b8e9493cecc1b6a085cecbc61b37179ff81acb5103" Oct 30 21:24:50.140075 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:50.140056 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" Oct 30 21:24:50.151398 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:50.151378 4848 scope.go:122] "RemoveContainer" containerID="353971ad64b4eee065b9e6a70897017bc4cd15f08639b81452e92b2a7de0d4a1" Oct 30 21:24:50.161382 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:50.161361 4848 scope.go:122] "RemoveContainer" containerID="3bbee5d997338eea519f63b8e9493cecc1b6a085cecbc61b37179ff81acb5103" Oct 30 21:24:50.161707 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:24:50.161688 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bbee5d997338eea519f63b8e9493cecc1b6a085cecbc61b37179ff81acb5103\": container with ID starting with 3bbee5d997338eea519f63b8e9493cecc1b6a085cecbc61b37179ff81acb5103 not found: ID does not exist" containerID="3bbee5d997338eea519f63b8e9493cecc1b6a085cecbc61b37179ff81acb5103" Oct 30 21:24:50.161868 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:50.161715 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bbee5d997338eea519f63b8e9493cecc1b6a085cecbc61b37179ff81acb5103"} err="failed to get container status \"3bbee5d997338eea519f63b8e9493cecc1b6a085cecbc61b37179ff81acb5103\": rpc error: code = NotFound desc = could not find container \"3bbee5d997338eea519f63b8e9493cecc1b6a085cecbc61b37179ff81acb5103\": container with ID starting with 3bbee5d997338eea519f63b8e9493cecc1b6a085cecbc61b37179ff81acb5103 not found: ID does not exist" Oct 30 21:24:50.161868 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:50.161734 4848 scope.go:122] "RemoveContainer" containerID="353971ad64b4eee065b9e6a70897017bc4cd15f08639b81452e92b2a7de0d4a1" Oct 30 21:24:50.162048 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:24:50.162029 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"353971ad64b4eee065b9e6a70897017bc4cd15f08639b81452e92b2a7de0d4a1\": container with ID starting with 353971ad64b4eee065b9e6a70897017bc4cd15f08639b81452e92b2a7de0d4a1 not found: ID does not exist" containerID="353971ad64b4eee065b9e6a70897017bc4cd15f08639b81452e92b2a7de0d4a1" Oct 30 21:24:50.162093 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:50.162055 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"353971ad64b4eee065b9e6a70897017bc4cd15f08639b81452e92b2a7de0d4a1"} err="failed to get container status \"353971ad64b4eee065b9e6a70897017bc4cd15f08639b81452e92b2a7de0d4a1\": rpc error: code = NotFound desc = could not find container \"353971ad64b4eee065b9e6a70897017bc4cd15f08639b81452e92b2a7de0d4a1\": container with ID starting with 353971ad64b4eee065b9e6a70897017bc4cd15f08639b81452e92b2a7de0d4a1 not found: ID does not exist" Oct 30 21:24:50.166234 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:50.166213 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x"] Oct 30 21:24:50.170599 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:50.170575 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a574c2-openshift-default-5c5f754885-hlc4x"] Oct 30 21:24:51.098792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:51.098679 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05d85436-702d-4fb3-8e98-220cb26ef773" path="/var/lib/kubelet/pods/05d85436-702d-4fb3-8e98-220cb26ef773/volumes" Oct 30 21:24:52.513418 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:24:52.513380 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:24:56.171235 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:24:56.171206 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" Oct 30 21:25:14.628427 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:14.628339 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9"] Oct 30 21:25:14.629029 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:14.628631 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" podUID="b0fca980-3e69-481f-a119-c4e4c2d56d5c" containerName="maas-api" containerID="cri-o://fe2422904974da9b08a24819ac68b739c5cac545279b83e57560557bfa9ccd26" gracePeriod=30 Oct 30 21:25:15.207204 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.207171 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" Oct 30 21:25:15.322581 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.322548 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0fca980-3e69-481f-a119-c4e4c2d56d5c-kube-api-access-wbwnn\" (UniqueName: \"kubernetes.io/projected/b0fca980-3e69-481f-a119-c4e4c2d56d5c-kube-api-access-wbwnn\") pod \"b0fca980-3e69-481f-a119-c4e4c2d56d5c\" (UID: \"b0fca980-3e69-481f-a119-c4e4c2d56d5c\") " Oct 30 21:25:15.322757 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.322641 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0fca980-3e69-481f-a119-c4e4c2d56d5c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fca980-3e69-481f-a119-c4e4c2d56d5c-maas-api-tls\") pod \"b0fca980-3e69-481f-a119-c4e4c2d56d5c\" (UID: \"b0fca980-3e69-481f-a119-c4e4c2d56d5c\") " Oct 30 21:25:15.322854 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.322841 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0fca980-3e69-481f-a119-c4e4c2d56d5c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fca980-3e69-481f-a119-c4e4c2d56d5c-openshift-service-ca\") pod \"b0fca980-3e69-481f-a119-c4e4c2d56d5c\" (UID: \"b0fca980-3e69-481f-a119-c4e4c2d56d5c\") " Oct 30 21:25:15.322973 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.322954 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0fca980-3e69-481f-a119-c4e4c2d56d5c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fca980-3e69-481f-a119-c4e4c2d56d5c-metrics-certs\") pod \"b0fca980-3e69-481f-a119-c4e4c2d56d5c\" (UID: \"b0fca980-3e69-481f-a119-c4e4c2d56d5c\") " Oct 30 21:25:15.323178 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.323152 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0fca980-3e69-481f-a119-c4e4c2d56d5c-openshift-service-ca" pod "b0fca980-3e69-481f-a119-c4e4c2d56d5c" (UID: "b0fca980-3e69-481f-a119-c4e4c2d56d5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:15.323552 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.323537 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fca980-3e69-481f-a119-c4e4c2d56d5c-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:15.325027 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.325005 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0fca980-3e69-481f-a119-c4e4c2d56d5c-metrics-certs" pod "b0fca980-3e69-481f-a119-c4e4c2d56d5c" (UID: "b0fca980-3e69-481f-a119-c4e4c2d56d5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:25:15.325207 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.325183 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0fca980-3e69-481f-a119-c4e4c2d56d5c-maas-api-tls" pod "b0fca980-3e69-481f-a119-c4e4c2d56d5c" (UID: "b0fca980-3e69-481f-a119-c4e4c2d56d5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:25:15.325207 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.325198 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0fca980-3e69-481f-a119-c4e4c2d56d5c-kube-api-access-wbwnn" pod "b0fca980-3e69-481f-a119-c4e4c2d56d5c" (UID: "b0fca980-3e69-481f-a119-c4e4c2d56d5c"). InnerVolumeSpecName "kube-api-access-wbwnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:15.424599 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.424573 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbwnn\" (UniqueName: \"kubernetes.io/projected/b0fca980-3e69-481f-a119-c4e4c2d56d5c-kube-api-access-wbwnn\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:15.424752 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.424602 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fca980-3e69-481f-a119-c4e4c2d56d5c-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:15.424752 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.424615 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fca980-3e69-481f-a119-c4e4c2d56d5c-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:15.452716 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.452674 4848 generic.go:356] "Generic (PLEG): container finished" podID="b0fca980-3e69-481f-a119-c4e4c2d56d5c" containerID="fe2422904974da9b08a24819ac68b739c5cac545279b83e57560557bfa9ccd26" exitCode=0 Oct 30 21:25:15.452716 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.452718 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" Oct 30 21:25:15.453005 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.452757 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" event={"ID":"b0fca980-3e69-481f-a119-c4e4c2d56d5c","Type":"ContainerDied","Data":"fe2422904974da9b08a24819ac68b739c5cac545279b83e57560557bfa9ccd26"} Oct 30 21:25:15.453005 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.452855 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9" event={"ID":"b0fca980-3e69-481f-a119-c4e4c2d56d5c","Type":"ContainerDied","Data":"be09eea135ec163cab029b9ab016755a362f7f4944d1a2e881ae4dc859c0054f"} Oct 30 21:25:15.453005 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.452878 4848 scope.go:122] "RemoveContainer" containerID="fe2422904974da9b08a24819ac68b739c5cac545279b83e57560557bfa9ccd26" Oct 30 21:25:15.464134 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.464114 4848 scope.go:122] "RemoveContainer" containerID="fe2422904974da9b08a24819ac68b739c5cac545279b83e57560557bfa9ccd26" Oct 30 21:25:15.464446 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:25:15.464414 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe2422904974da9b08a24819ac68b739c5cac545279b83e57560557bfa9ccd26\": container with ID starting with fe2422904974da9b08a24819ac68b739c5cac545279b83e57560557bfa9ccd26 not found: ID does not exist" containerID="fe2422904974da9b08a24819ac68b739c5cac545279b83e57560557bfa9ccd26" Oct 30 21:25:15.464544 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.464456 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe2422904974da9b08a24819ac68b739c5cac545279b83e57560557bfa9ccd26"} err="failed to get container status \"fe2422904974da9b08a24819ac68b739c5cac545279b83e57560557bfa9ccd26\": rpc error: code = NotFound desc = could not find container \"fe2422904974da9b08a24819ac68b739c5cac545279b83e57560557bfa9ccd26\": container with ID starting with fe2422904974da9b08a24819ac68b739c5cac545279b83e57560557bfa9ccd26 not found: ID does not exist" Oct 30 21:25:15.477714 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.477692 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9"] Oct 30 21:25:15.485893 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:15.485871 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-176db5-5f4564c5b7-98tf9"] Oct 30 21:25:16.933579 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:16.933540 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0fca980-3e69-481f-a119-c4e4c2d56d5c" path="/var/lib/kubelet/pods/b0fca980-3e69-481f-a119-c4e4c2d56d5c/volumes" Oct 30 21:25:22.510788 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:25:22.510738 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:25:23.336370 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:23.336334 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9"] Oct 30 21:25:23.336663 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:23.336635 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" podUID="50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" containerName="istio-proxy" containerID="cri-o://9d59451822af795ebb75427a624b3e7dd8a75a466cdf648109eb1cc01e232139" gracePeriod=30 Oct 30 21:25:24.039737 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:24.039703 4848 patch_prober.go:29] interesting pod/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:25:24.040175 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:24.039780 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" podUID="50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:25:28.575336 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.575295 4848 generic.go:356] "Generic (PLEG): container finished" podID="50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" containerID="9d59451822af795ebb75427a624b3e7dd8a75a466cdf648109eb1cc01e232139" exitCode=0 Oct 30 21:25:28.575819 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.575500 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" event={"ID":"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8","Type":"ContainerDied","Data":"9d59451822af795ebb75427a624b3e7dd8a75a466cdf648109eb1cc01e232139"} Oct 30 21:25:28.575819 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.575596 4848 scope.go:122] "RemoveContainer" containerID="1d8f8d4ca07607946b6bd623c37e21cceed8ec27b5bc91aea6e27c4d97e51d13" Oct 30 21:25:28.641715 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.641686 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:25:28.736984 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.736949 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-token\" (UniqueName: \"kubernetes.io/projected/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-token\") pod \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " Oct 30 21:25:28.737201 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.736998 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-kube-api-access-5j9f5\" (UniqueName: \"kubernetes.io/projected/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-kube-api-access-5j9f5\") pod \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " Oct 30 21:25:28.737201 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.737065 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istiod-ca-cert\") pod \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " Oct 30 21:25:28.737201 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.737098 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-podinfo\") pod \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " Oct 30 21:25:28.737201 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.737127 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-workload-certs\") pod \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " Oct 30 21:25:28.737201 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.737160 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-data\") pod \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " Oct 30 21:25:28.737485 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.737269 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-credential-socket\") pod \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " Oct 30 21:25:28.737485 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.737301 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-envoy\") pod \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " Oct 30 21:25:28.737485 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.737387 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-workload-socket\") pod \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\" (UID: \"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8\") " Oct 30 21:25:28.738352 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.738256 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-credential-socket" pod "50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" (UID: "50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:28.738352 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.738279 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-data" pod "50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" (UID: "50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:28.738889 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.738865 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-workload-socket" pod "50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" (UID: "50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:28.739064 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.738882 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-workload-certs" pod "50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" (UID: "50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:28.739339 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.739229 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istiod-ca-cert" pod "50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" (UID: "50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:28.740286 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.740261 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-kube-api-access-5j9f5" pod "50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" (UID: "50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8"). InnerVolumeSpecName "kube-api-access-5j9f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:28.740381 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.740335 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-podinfo" pod "50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" (UID: "50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:25:28.741457 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.741418 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-token" pod "50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" (UID: "50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:28.748918 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.748874 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-envoy" pod "50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" (UID: "50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:28.839630 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.839594 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:28.839630 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.839625 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:28.839630 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.839635 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:28.840187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.839643 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:28.840187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.839652 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:28.840187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.839661 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:28.840187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.839669 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:28.840187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.839677 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:28.840187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:28.839685 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j9f5\" (UniqueName: \"kubernetes.io/projected/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8-kube-api-access-5j9f5\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:29.583432 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:29.583399 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" Oct 30 21:25:29.583891 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:29.583438 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9" event={"ID":"50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8","Type":"ContainerDied","Data":"997baa0be6e5a9cb9a1000c772ce8ad14a6bb86506f2daa8f9487059f88061c2"} Oct 30 21:25:29.583891 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:29.583497 4848 scope.go:122] "RemoveContainer" containerID="9d59451822af795ebb75427a624b3e7dd8a75a466cdf648109eb1cc01e232139" Oct 30 21:25:29.604906 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:29.604884 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9"] Oct 30 21:25:29.609917 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:29.609893 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-176db5-openshift-default-76d7ddc765-x6dh9"] Oct 30 21:25:30.933749 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:30.933717 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" path="/var/lib/kubelet/pods/50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8/volumes" Oct 30 21:25:41.679410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.679352 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6"] Oct 30 21:25:41.680386 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.680362 4848 cpu_manager.go:414] "containerMap: removing container" podUID="50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" containerName="istio-proxy" Oct 30 21:25:41.680386 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.680385 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" containerName="istio-proxy" Oct 30 21:25:41.680586 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.680420 4848 cpu_manager.go:414] "containerMap: removing container" podUID="05d85436-702d-4fb3-8e98-220cb26ef773" containerName="istio-proxy" Oct 30 21:25:41.680586 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.680428 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d85436-702d-4fb3-8e98-220cb26ef773" containerName="istio-proxy" Oct 30 21:25:41.680586 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.680449 4848 cpu_manager.go:414] "containerMap: removing container" podUID="b0fca980-3e69-481f-a119-c4e4c2d56d5c" containerName="maas-api" Oct 30 21:25:41.680586 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.680456 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0fca980-3e69-481f-a119-c4e4c2d56d5c" containerName="maas-api" Oct 30 21:25:41.680586 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.680464 4848 cpu_manager.go:414] "containerMap: removing container" podUID="05d85436-702d-4fb3-8e98-220cb26ef773" containerName="istio-proxy" Oct 30 21:25:41.680586 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.680470 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d85436-702d-4fb3-8e98-220cb26ef773" containerName="istio-proxy" Oct 30 21:25:41.680971 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.680629 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d85436-702d-4fb3-8e98-220cb26ef773" containerName="istio-proxy" Oct 30 21:25:41.680971 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.680645 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0fca980-3e69-481f-a119-c4e4c2d56d5c" containerName="maas-api" Oct 30 21:25:41.680971 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.680653 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d85436-702d-4fb3-8e98-220cb26ef773" containerName="istio-proxy" Oct 30 21:25:41.680971 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.680667 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" containerName="istio-proxy" Oct 30 21:25:41.680971 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.680680 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" containerName="istio-proxy" Oct 30 21:25:41.691871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.685630 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:41.694858 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.694827 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6"] Oct 30 21:25:41.695335 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.695311 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2ac526\"/\"test-model-2ac526-kserve-self-signed-certs\"" Oct 30 21:25:41.695491 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.695450 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2ac526\"/\"default-dockercfg-6gwbd\"" Oct 30 21:25:41.695576 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.695503 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2ac526\"/\"kube-root-ca.crt\"" Oct 30 21:25:41.696437 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.695974 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2ac526\"/\"openshift-service-ca.crt\"" Oct 30 21:25:41.700976 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.700259 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7hmx\" (UniqueName: \"kubernetes.io/projected/24daadef-9dca-41c5-876f-9210ce533955-kube-api-access-t7hmx\") pod \"test-model-2ac526-kserve-78699d86c7-ptnf6\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:41.700976 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.700568 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-tmp-dir\") pod \"test-model-2ac526-kserve-78699d86c7-ptnf6\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:41.700976 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.700631 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-dshm\") pod \"test-model-2ac526-kserve-78699d86c7-ptnf6\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:41.700976 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.700658 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-model-cache\") pod \"test-model-2ac526-kserve-78699d86c7-ptnf6\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:41.700976 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.700779 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-home\") pod \"test-model-2ac526-kserve-78699d86c7-ptnf6\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:41.700976 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.700821 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24daadef-9dca-41c5-876f-9210ce533955-tls-certs\") pod \"test-model-2ac526-kserve-78699d86c7-ptnf6\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:41.802663 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.802620 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-dshm\") pod \"test-model-2ac526-kserve-78699d86c7-ptnf6\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:41.802663 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.802661 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-model-cache\") pod \"test-model-2ac526-kserve-78699d86c7-ptnf6\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:41.802967 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.802744 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-home\") pod \"test-model-2ac526-kserve-78699d86c7-ptnf6\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:41.802967 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.802792 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24daadef-9dca-41c5-876f-9210ce533955-tls-certs\") pod \"test-model-2ac526-kserve-78699d86c7-ptnf6\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:41.802967 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.802815 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7hmx\" (UniqueName: \"kubernetes.io/projected/24daadef-9dca-41c5-876f-9210ce533955-kube-api-access-t7hmx\") pod \"test-model-2ac526-kserve-78699d86c7-ptnf6\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:41.803149 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.803132 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-model-cache\") pod \"test-model-2ac526-kserve-78699d86c7-ptnf6\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:41.803640 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.803213 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-home\") pod \"test-model-2ac526-kserve-78699d86c7-ptnf6\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:41.803640 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.803509 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-tmp-dir\") pod \"test-model-2ac526-kserve-78699d86c7-ptnf6\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:41.804374 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.804351 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-tmp-dir\") pod \"test-model-2ac526-kserve-78699d86c7-ptnf6\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:41.805645 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.805618 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-dshm\") pod \"test-model-2ac526-kserve-78699d86c7-ptnf6\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:41.805941 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.805923 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24daadef-9dca-41c5-876f-9210ce533955-tls-certs\") pod \"test-model-2ac526-kserve-78699d86c7-ptnf6\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:41.812219 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:41.812197 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7hmx\" (UniqueName: \"kubernetes.io/projected/24daadef-9dca-41c5-876f-9210ce533955-kube-api-access-t7hmx\") pod \"test-model-2ac526-kserve-78699d86c7-ptnf6\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:42.007581 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:42.007502 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:42.320103 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:25:42.320074 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24daadef_9dca_41c5_876f_9210ce533955.slice/crio-a106395c24daa501e0d91b6c03eca8ed4f3adc747905663af4926472059ef9bf WatchSource:0}: Error finding container a106395c24daa501e0d91b6c03eca8ed4f3adc747905663af4926472059ef9bf: Status 404 returned error can't find the container with id a106395c24daa501e0d91b6c03eca8ed4f3adc747905663af4926472059ef9bf Oct 30 21:25:42.320371 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:42.320345 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6"] Oct 30 21:25:42.769296 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:42.769249 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" event={"ID":"24daadef-9dca-41c5-876f-9210ce533955","Type":"ContainerStarted","Data":"fe07862ce66ec71411925c6d6b725e76a045527023f09cb8b07aeab769e23ca8"} Oct 30 21:25:42.769714 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:42.769326 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" event={"ID":"24daadef-9dca-41c5-876f-9210ce533955","Type":"ContainerStarted","Data":"a106395c24daa501e0d91b6c03eca8ed4f3adc747905663af4926472059ef9bf"} Oct 30 21:25:42.807052 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:42.806779 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" podStartSLOduration=1.8067463849999998 podStartE2EDuration="1.806746385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:42.804295625 +0000 UTC m=+2847.984192132" watchObservedRunningTime="2026-10-30 21:25:42.806746385 +0000 UTC m=+2847.986642893" Oct 30 21:25:43.008683 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:43.008647 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:44.785361 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:44.785329 4848 generic.go:356] "Generic (PLEG): container finished" podID="b02fca84-f078-4d46-a842-a9b29f63eaaa" containerID="0a84b750ea01dadef8d7b57baf99397ec6cfe24d43c7e17c771770418df3b565" exitCode=0 Oct 30 21:25:44.785960 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:44.785401 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" event={"ID":"b02fca84-f078-4d46-a842-a9b29f63eaaa","Type":"ContainerDied","Data":"0a84b750ea01dadef8d7b57baf99397ec6cfe24d43c7e17c771770418df3b565"} Oct 30 21:25:44.786048 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:44.786012 4848 scope.go:122] "RemoveContainer" containerID="0a84b750ea01dadef8d7b57baf99397ec6cfe24d43c7e17c771770418df3b565" Oct 30 21:25:44.787837 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:44.787810 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:25:44.787936 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:44.787878 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:25:44.787936 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:44.787926 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:25:45.795090 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:45.795052 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" event={"ID":"b02fca84-f078-4d46-a842-a9b29f63eaaa","Type":"ContainerStarted","Data":"4ce63e6a98d91d742876bc882581170a7fc53b000a45b9fdc6e028d47cd7d7c9"} Oct 30 21:25:45.993860 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:45.993014 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p"] Oct 30 21:25:45.993860 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:45.993276 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" podUID="8f3a05c8-cf06-4bc9-9131-8c3cdf20811f" containerName="maas-api" containerID="cri-o://3fa7123a118cfac148090652202418fcf0a0cc15402776318d69943aadda9862" gracePeriod=30 Oct 30 21:25:46.137347 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.137316 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:25:46.138531 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.138506 4848 patch_prober.go:29] interesting pod/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 21:25:46.138623 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.138552 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" podUID="b02fca84-f078-4d46-a842-a9b29f63eaaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 21:25:46.744521 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.744484 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" Oct 30 21:25:46.780077 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.780046 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-maas-api-tls\") pod \"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f\" (UID: \"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f\") " Oct 30 21:25:46.780271 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.780215 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-kube-api-access-nkg66\" (UniqueName: \"kubernetes.io/projected/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-kube-api-access-nkg66\") pod \"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f\" (UID: \"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f\") " Oct 30 21:25:46.780370 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.780331 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-metrics-certs\") pod \"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f\" (UID: \"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f\") " Oct 30 21:25:46.780435 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.780415 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-openshift-service-ca\") pod \"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f\" (UID: \"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f\") " Oct 30 21:25:46.780974 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.780941 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-openshift-service-ca" pod "8f3a05c8-cf06-4bc9-9131-8c3cdf20811f" (UID: "8f3a05c8-cf06-4bc9-9131-8c3cdf20811f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:46.782609 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.782584 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-kube-api-access-nkg66" pod "8f3a05c8-cf06-4bc9-9131-8c3cdf20811f" (UID: "8f3a05c8-cf06-4bc9-9131-8c3cdf20811f"). InnerVolumeSpecName "kube-api-access-nkg66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:46.782729 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.782685 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-metrics-certs" pod "8f3a05c8-cf06-4bc9-9131-8c3cdf20811f" (UID: "8f3a05c8-cf06-4bc9-9131-8c3cdf20811f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:25:46.783247 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.783214 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-maas-api-tls" pod "8f3a05c8-cf06-4bc9-9131-8c3cdf20811f" (UID: "8f3a05c8-cf06-4bc9-9131-8c3cdf20811f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:25:46.835625 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.835585 4848 generic.go:356] "Generic (PLEG): container finished" podID="8f3a05c8-cf06-4bc9-9131-8c3cdf20811f" containerID="3fa7123a118cfac148090652202418fcf0a0cc15402776318d69943aadda9862" exitCode=0 Oct 30 21:25:46.836914 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.836892 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" Oct 30 21:25:46.838176 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.838142 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" event={"ID":"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f","Type":"ContainerDied","Data":"3fa7123a118cfac148090652202418fcf0a0cc15402776318d69943aadda9862"} Oct 30 21:25:46.838310 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.838208 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p" event={"ID":"8f3a05c8-cf06-4bc9-9131-8c3cdf20811f","Type":"ContainerDied","Data":"65c6fa491b226174e4c4303da645a0b7eb4e70e88e94b6a3f80e78a24b67f975"} Oct 30 21:25:46.838310 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.838241 4848 scope.go:122] "RemoveContainer" containerID="3fa7123a118cfac148090652202418fcf0a0cc15402776318d69943aadda9862" Oct 30 21:25:46.850551 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.850530 4848 scope.go:122] "RemoveContainer" containerID="3fa7123a118cfac148090652202418fcf0a0cc15402776318d69943aadda9862" Oct 30 21:25:46.850844 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:25:46.850824 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa7123a118cfac148090652202418fcf0a0cc15402776318d69943aadda9862\": container with ID starting with 3fa7123a118cfac148090652202418fcf0a0cc15402776318d69943aadda9862 not found: ID does not exist" containerID="3fa7123a118cfac148090652202418fcf0a0cc15402776318d69943aadda9862" Oct 30 21:25:46.850904 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.850854 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa7123a118cfac148090652202418fcf0a0cc15402776318d69943aadda9862"} err="failed to get container status \"3fa7123a118cfac148090652202418fcf0a0cc15402776318d69943aadda9862\": rpc error: code = NotFound desc = could not find container \"3fa7123a118cfac148090652202418fcf0a0cc15402776318d69943aadda9862\": container with ID starting with 3fa7123a118cfac148090652202418fcf0a0cc15402776318d69943aadda9862 not found: ID does not exist" Oct 30 21:25:46.864952 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.864685 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p"] Oct 30 21:25:46.869012 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.868988 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-74f9f4-c94fb6b5-p6t6p"] Oct 30 21:25:46.914791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.885526 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:46.914791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.885573 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:46.914791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.885593 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:46.914791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.885607 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkg66\" (UniqueName: \"kubernetes.io/projected/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f-kube-api-access-nkg66\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:46.977689 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:46.977611 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f3a05c8-cf06-4bc9-9131-8c3cdf20811f" path="/var/lib/kubelet/pods/8f3a05c8-cf06-4bc9-9131-8c3cdf20811f/volumes" Oct 30 21:25:47.137511 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:47.137477 4848 patch_prober.go:29] interesting pod/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 21:25:47.137679 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:47.137530 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" podUID="b02fca84-f078-4d46-a842-a9b29f63eaaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 21:25:48.126789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:48.123881 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:25:48.140790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:48.140175 4848 patch_prober.go:29] interesting pod/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 21:25:48.140790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:48.140240 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" podUID="b02fca84-f078-4d46-a842-a9b29f63eaaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 21:25:49.137193 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:49.137156 4848 patch_prober.go:29] interesting pod/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 21:25:49.137613 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:49.137211 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" podUID="b02fca84-f078-4d46-a842-a9b29f63eaaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 21:25:50.137442 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:50.137408 4848 patch_prober.go:29] interesting pod/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 21:25:50.137887 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:50.137460 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" podUID="b02fca84-f078-4d46-a842-a9b29f63eaaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 21:25:51.137266 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:51.137226 4848 patch_prober.go:29] interesting pod/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 21:25:51.137445 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:51.137284 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" podUID="b02fca84-f078-4d46-a842-a9b29f63eaaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 21:25:52.008172 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.008131 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:52.020872 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.020844 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:52.142341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.142309 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:25:52.143393 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.143371 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:25:52.379951 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.379877 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x"] Oct 30 21:25:52.380721 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.380701 4848 cpu_manager.go:414] "containerMap: removing container" podUID="50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" containerName="istio-proxy" Oct 30 21:25:52.380721 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.380722 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f3e30a-b03e-4b05-b4e0-06a1d1ef40f8" containerName="istio-proxy" Oct 30 21:25:52.380964 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.380748 4848 cpu_manager.go:414] "containerMap: removing container" podUID="8f3a05c8-cf06-4bc9-9131-8c3cdf20811f" containerName="maas-api" Oct 30 21:25:52.380964 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.380756 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f3a05c8-cf06-4bc9-9131-8c3cdf20811f" containerName="maas-api" Oct 30 21:25:52.381077 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.381012 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f3a05c8-cf06-4bc9-9131-8c3cdf20811f" containerName="maas-api" Oct 30 21:25:52.384962 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.384931 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:52.394895 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.394867 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x"] Oct 30 21:25:52.397718 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.397692 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a3fb70\"/\"openshift-service-ca.crt\"" Oct 30 21:25:52.397872 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.397750 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a3fb70\"/\"default-dockercfg-rgzwz\"" Oct 30 21:25:52.397872 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.397860 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a3fb70\"/\"kube-root-ca.crt\"" Oct 30 21:25:52.398002 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.397929 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a3fb70\"/\"test-model-a3fb70-kserve-self-signed-certs\"" Oct 30 21:25:52.497525 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.497494 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcws2\" (UniqueName: \"kubernetes.io/projected/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-kube-api-access-dcws2\") pod \"test-model-a3fb70-kserve-7789b69865-4m55x\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:52.497741 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.497589 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-tmp-dir\") pod \"test-model-a3fb70-kserve-7789b69865-4m55x\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:52.497836 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.497808 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-home\") pod \"test-model-a3fb70-kserve-7789b69865-4m55x\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:52.497913 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.497889 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-dshm\") pod \"test-model-a3fb70-kserve-7789b69865-4m55x\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:52.498127 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.498107 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-model-cache\") pod \"test-model-a3fb70-kserve-7789b69865-4m55x\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:52.498258 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.498176 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-tls-certs\") pod \"test-model-a3fb70-kserve-7789b69865-4m55x\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:52.518619 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:25:52.518583 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:25:52.606809 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.602151 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-model-cache\") pod \"test-model-a3fb70-kserve-7789b69865-4m55x\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:52.607150 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.607124 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-tls-certs\") pod \"test-model-a3fb70-kserve-7789b69865-4m55x\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:52.607356 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.607339 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcws2\" (UniqueName: \"kubernetes.io/projected/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-kube-api-access-dcws2\") pod \"test-model-a3fb70-kserve-7789b69865-4m55x\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:52.607495 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.607459 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-tmp-dir\") pod \"test-model-a3fb70-kserve-7789b69865-4m55x\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:52.607794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.607751 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-home\") pod \"test-model-a3fb70-kserve-7789b69865-4m55x\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:52.608006 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.607987 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-dshm\") pod \"test-model-a3fb70-kserve-7789b69865-4m55x\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:52.609835 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.609091 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-model-cache\") pod \"test-model-a3fb70-kserve-7789b69865-4m55x\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:52.609835 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.609350 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-tmp-dir\") pod \"test-model-a3fb70-kserve-7789b69865-4m55x\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:52.609835 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.609585 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-home\") pod \"test-model-a3fb70-kserve-7789b69865-4m55x\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:52.671838 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.640526 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcws2\" (UniqueName: \"kubernetes.io/projected/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-kube-api-access-dcws2\") pod \"test-model-a3fb70-kserve-7789b69865-4m55x\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:52.702796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.702391 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-dshm\") pod \"test-model-a3fb70-kserve-7789b69865-4m55x\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:52.715826 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.713462 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-tls-certs\") pod \"test-model-a3fb70-kserve-7789b69865-4m55x\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:52.897918 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:52.897888 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:25:53.012748 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:53.012715 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:53.303330 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:25:53.303301 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57e19834_f4ba_495f_a5a8_2b73f8e36cb8.slice/crio-1f47a6dd20994aba35e523876226c201028012fb72dc2a07b2139a2a35dbe77f WatchSource:0}: Error finding container 1f47a6dd20994aba35e523876226c201028012fb72dc2a07b2139a2a35dbe77f: Status 404 returned error can't find the container with id 1f47a6dd20994aba35e523876226c201028012fb72dc2a07b2139a2a35dbe77f Oct 30 21:25:53.303529 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:53.303505 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x"] Oct 30 21:25:53.896715 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:53.896678 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" event={"ID":"57e19834-f4ba-495f-a5a8-2b73f8e36cb8","Type":"ContainerStarted","Data":"8d4fec84f2979e4dc9e66d392b8afac418abfdbee69735c3e96275f75f91f4e1"} Oct 30 21:25:53.896930 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:53.896724 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" event={"ID":"57e19834-f4ba-495f-a5a8-2b73f8e36cb8","Type":"ContainerStarted","Data":"1f47a6dd20994aba35e523876226c201028012fb72dc2a07b2139a2a35dbe77f"} Oct 30 21:25:53.920394 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:53.920346 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" podStartSLOduration=1.920329432 podStartE2EDuration="1.920329432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:53.91666233 +0000 UTC m=+2859.096558831" watchObservedRunningTime="2026-10-30 21:25:53.920329432 +0000 UTC m=+2859.100225940" Oct 30 21:25:54.013329 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:54.013287 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:25:54.225196 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:54.225154 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27"] Oct 30 21:25:54.225435 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:54.225408 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" podUID="dcfcc9ac-8303-4921-b9f8-f7654af44897" containerName="istio-proxy" containerID="cri-o://79acc039d2cd9635e39a5b6269b2006cdeb360c542eb1c8a5be55a76ba1ff256" gracePeriod=30 Oct 30 21:25:59.733430 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.733389 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g"] Oct 30 21:25:59.792976 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.792941 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:25:59.928723 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.928685 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-credential-socket\") pod \"dcfcc9ac-8303-4921-b9f8-f7654af44897\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " Oct 30 21:25:59.928979 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.928747 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcfcc9ac-8303-4921-b9f8-f7654af44897-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcfcc9ac-8303-4921-b9f8-f7654af44897-istiod-ca-cert\") pod \"dcfcc9ac-8303-4921-b9f8-f7654af44897\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " Oct 30 21:25:59.928979 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.928797 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcfcc9ac-8303-4921-b9f8-f7654af44897-kube-api-access-jb4dl\" (UniqueName: \"kubernetes.io/projected/dcfcc9ac-8303-4921-b9f8-f7654af44897-kube-api-access-jb4dl\") pod \"dcfcc9ac-8303-4921-b9f8-f7654af44897\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " Oct 30 21:25:59.928979 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.928902 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-token\" (UniqueName: \"kubernetes.io/projected/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-token\") pod \"dcfcc9ac-8303-4921-b9f8-f7654af44897\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " Oct 30 21:25:59.928979 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.928974 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-podinfo\") pod \"dcfcc9ac-8303-4921-b9f8-f7654af44897\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " Oct 30 21:25:59.929259 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.929044 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-credential-socket" pod "dcfcc9ac-8303-4921-b9f8-f7654af44897" (UID: "dcfcc9ac-8303-4921-b9f8-f7654af44897"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.929259 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.929119 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-envoy\") pod \"dcfcc9ac-8303-4921-b9f8-f7654af44897\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " Oct 30 21:25:59.929259 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.929191 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-workload-certs\") pod \"dcfcc9ac-8303-4921-b9f8-f7654af44897\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " Oct 30 21:25:59.929259 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.929228 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcfcc9ac-8303-4921-b9f8-f7654af44897-istiod-ca-cert" pod "dcfcc9ac-8303-4921-b9f8-f7654af44897" (UID: "dcfcc9ac-8303-4921-b9f8-f7654af44897"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:59.929259 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.929256 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-workload-socket\") pod \"dcfcc9ac-8303-4921-b9f8-f7654af44897\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " Oct 30 21:25:59.929465 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.929293 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-data\") pod \"dcfcc9ac-8303-4921-b9f8-f7654af44897\" (UID: \"dcfcc9ac-8303-4921-b9f8-f7654af44897\") " Oct 30 21:25:59.929465 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.929427 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-workload-certs" pod "dcfcc9ac-8303-4921-b9f8-f7654af44897" (UID: "dcfcc9ac-8303-4921-b9f8-f7654af44897"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.929602 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.929581 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-workload-socket" pod "dcfcc9ac-8303-4921-b9f8-f7654af44897" (UID: "dcfcc9ac-8303-4921-b9f8-f7654af44897"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.929719 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.929693 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-data" pod "dcfcc9ac-8303-4921-b9f8-f7654af44897" (UID: "dcfcc9ac-8303-4921-b9f8-f7654af44897"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.929914 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.929894 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.929994 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.929920 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcfcc9ac-8303-4921-b9f8-f7654af44897-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.929994 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.929934 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.929994 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.929947 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.929994 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.929960 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:25:59.931436 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.931415 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-podinfo" pod "dcfcc9ac-8303-4921-b9f8-f7654af44897" (UID: "dcfcc9ac-8303-4921-b9f8-f7654af44897"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:25:59.931664 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.931643 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcfcc9ac-8303-4921-b9f8-f7654af44897-kube-api-access-jb4dl" pod "dcfcc9ac-8303-4921-b9f8-f7654af44897" (UID: "dcfcc9ac-8303-4921-b9f8-f7654af44897"). InnerVolumeSpecName "kube-api-access-jb4dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:59.931917 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.931902 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-envoy" pod "dcfcc9ac-8303-4921-b9f8-f7654af44897" (UID: "dcfcc9ac-8303-4921-b9f8-f7654af44897"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:25:59.931984 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.931953 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-token" pod "dcfcc9ac-8303-4921-b9f8-f7654af44897" (UID: "dcfcc9ac-8303-4921-b9f8-f7654af44897"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:59.951200 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.951169 4848 generic.go:356] "Generic (PLEG): container finished" podID="dcfcc9ac-8303-4921-b9f8-f7654af44897" containerID="79acc039d2cd9635e39a5b6269b2006cdeb360c542eb1c8a5be55a76ba1ff256" exitCode=0 Oct 30 21:25:59.951300 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.951249 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" event={"ID":"dcfcc9ac-8303-4921-b9f8-f7654af44897","Type":"ContainerDied","Data":"79acc039d2cd9635e39a5b6269b2006cdeb360c542eb1c8a5be55a76ba1ff256"} Oct 30 21:25:59.951358 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.951314 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" event={"ID":"dcfcc9ac-8303-4921-b9f8-f7654af44897","Type":"ContainerDied","Data":"237c06e0d6b84a6710a5d42bd569660a3bae9ab538c430ea6b3800002a764696"} Oct 30 21:25:59.951358 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.951257 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27" Oct 30 21:25:59.951358 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.951345 4848 scope.go:122] "RemoveContainer" containerID="79acc039d2cd9635e39a5b6269b2006cdeb360c542eb1c8a5be55a76ba1ff256" Oct 30 21:25:59.968103 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.968081 4848 scope.go:122] "RemoveContainer" containerID="79acc039d2cd9635e39a5b6269b2006cdeb360c542eb1c8a5be55a76ba1ff256" Oct 30 21:25:59.968437 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:25:59.968417 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79acc039d2cd9635e39a5b6269b2006cdeb360c542eb1c8a5be55a76ba1ff256\": container with ID starting with 79acc039d2cd9635e39a5b6269b2006cdeb360c542eb1c8a5be55a76ba1ff256 not found: ID does not exist" containerID="79acc039d2cd9635e39a5b6269b2006cdeb360c542eb1c8a5be55a76ba1ff256" Oct 30 21:25:59.968503 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.968450 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79acc039d2cd9635e39a5b6269b2006cdeb360c542eb1c8a5be55a76ba1ff256"} err="failed to get container status \"79acc039d2cd9635e39a5b6269b2006cdeb360c542eb1c8a5be55a76ba1ff256\": rpc error: code = NotFound desc = could not find container \"79acc039d2cd9635e39a5b6269b2006cdeb360c542eb1c8a5be55a76ba1ff256\": container with ID starting with 79acc039d2cd9635e39a5b6269b2006cdeb360c542eb1c8a5be55a76ba1ff256 not found: ID does not exist" Oct 30 21:25:59.988394 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.988307 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27"] Oct 30 21:25:59.992973 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:25:59.992946 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-74f9f4-openshift-default-5dcf868997-g7v27"] Oct 30 21:26:00.039788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:00.037700 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:26:00.039788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:00.037735 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:26:00.039788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:00.037752 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcfcc9ac-8303-4921-b9f8-f7654af44897-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:26:00.039788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:00.037783 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb4dl\" (UniqueName: \"kubernetes.io/projected/dcfcc9ac-8303-4921-b9f8-f7654af44897-kube-api-access-jb4dl\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:26:00.946151 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:00.946103 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcfcc9ac-8303-4921-b9f8-f7654af44897" path="/var/lib/kubelet/pods/dcfcc9ac-8303-4921-b9f8-f7654af44897/volumes" Oct 30 21:26:03.012997 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:03.012956 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:26:03.026070 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:03.026044 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:26:04.044271 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:04.044241 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:26:07.973119 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:07.973074 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4"] Oct 30 21:26:07.973854 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:07.973833 4848 cpu_manager.go:414] "containerMap: removing container" podUID="dcfcc9ac-8303-4921-b9f8-f7654af44897" containerName="istio-proxy" Oct 30 21:26:07.974076 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:07.974046 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcfcc9ac-8303-4921-b9f8-f7654af44897" containerName="istio-proxy" Oct 30 21:26:07.974286 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:07.974273 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcfcc9ac-8303-4921-b9f8-f7654af44897" containerName="istio-proxy" Oct 30 21:26:07.978269 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:07.978249 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:07.994724 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:07.994680 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0c4052-openshift-default-dockercfg-mt57s\"" Oct 30 21:26:07.996448 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:07.996421 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4"] Oct 30 21:26:08.134788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.133851 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-credential-socket\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.134788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.133903 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-workload-socket\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.134788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.133961 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-workload-certs\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.134788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.134029 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-token\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.134788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.134060 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istiod-ca-cert\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.134788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.134228 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-envoy\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.134788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.134253 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdh44\" (UniqueName: \"kubernetes.io/projected/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-kube-api-access-tdh44\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.134788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.134279 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-data\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.134788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.134330 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-podinfo\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.238853 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.238713 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-envoy\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.238853 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.238780 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdh44\" (UniqueName: \"kubernetes.io/projected/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-kube-api-access-tdh44\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.238853 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.238806 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-data\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.238853 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.238854 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-podinfo\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.239216 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.239006 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-credential-socket\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.239216 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.239038 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-workload-socket\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.239216 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.239082 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-workload-certs\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.239216 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.239149 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-token\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.239216 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.239176 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istiod-ca-cert\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.243821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.240150 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istiod-ca-cert\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.243821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.241086 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-credential-socket\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.243821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.241582 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-data\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.243821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.242154 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-workload-certs\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.243821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.242366 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-workload-socket\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.253790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.252378 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-podinfo\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.263595 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.263126 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-envoy\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.279274 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.274555 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdh44\" (UniqueName: \"kubernetes.io/projected/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-kube-api-access-tdh44\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.279274 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.279204 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-token\") pod \"e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.302404 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.301276 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:08.666889 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:26:08.666852 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1a3cb7_8aac_4c2c_8c6c_b2a7f4507c2d.slice/crio-1449687a00a94e1cd9deaaf0aeab9ae25d0a25ea75295c27991e8174721712fd WatchSource:0}: Error finding container 1449687a00a94e1cd9deaaf0aeab9ae25d0a25ea75295c27991e8174721712fd: Status 404 returned error can't find the container with id 1449687a00a94e1cd9deaaf0aeab9ae25d0a25ea75295c27991e8174721712fd Oct 30 21:26:08.669387 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.669358 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4"] Oct 30 21:26:08.670264 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.670227 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:08.670379 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.670338 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:08.670454 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:08.670383 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:09.155040 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:09.154988 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" event={"ID":"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d","Type":"ContainerStarted","Data":"00f4dc6a43893ba52fa8978eedb79498e9abbf613e28ca9223114d0bed393d67"} Oct 30 21:26:09.155441 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:09.155062 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" event={"ID":"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d","Type":"ContainerStarted","Data":"1449687a00a94e1cd9deaaf0aeab9ae25d0a25ea75295c27991e8174721712fd"} Oct 30 21:26:09.179992 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:09.179933 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" podStartSLOduration=2.179916341 podStartE2EDuration="2.179916341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:09.175839609 +0000 UTC m=+2874.355736116" watchObservedRunningTime="2026-10-30 21:26:09.179916341 +0000 UTC m=+2874.359812847" Oct 30 21:26:09.318792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:09.307299 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:09.341077 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:09.338489 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:10.162257 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:10.162224 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:10.163212 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:10.163195 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:26:12.880703 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:12.858027 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7"] Oct 30 21:26:12.910793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:12.901407 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7"] Oct 30 21:26:12.910793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:12.901547 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7" Oct 30 21:26:12.983254 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:12.973830 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0c4052\"" Oct 30 21:26:12.983254 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:12.974107 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0c4052-dockercfg-n4bdr\"" Oct 30 21:26:13.024173 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:13.013735 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt5bq\" (UniqueName: \"kubernetes.io/projected/fa84afb2-c58a-4ac6-be54-bdfef411fd2b-kube-api-access-mt5bq\") pod \"payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7\" (UID: \"fa84afb2-c58a-4ac6-be54-bdfef411fd2b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7" Oct 30 21:26:13.024578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:13.024553 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa84afb2-c58a-4ac6-be54-bdfef411fd2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7\" (UID: \"fa84afb2-c58a-4ac6-be54-bdfef411fd2b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7" Oct 30 21:26:13.136967 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:13.136876 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt5bq\" (UniqueName: \"kubernetes.io/projected/fa84afb2-c58a-4ac6-be54-bdfef411fd2b-kube-api-access-mt5bq\") pod \"payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7\" (UID: \"fa84afb2-c58a-4ac6-be54-bdfef411fd2b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7" Oct 30 21:26:13.137643 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:13.137619 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa84afb2-c58a-4ac6-be54-bdfef411fd2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7\" (UID: \"fa84afb2-c58a-4ac6-be54-bdfef411fd2b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7" Oct 30 21:26:13.170253 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:13.138617 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa84afb2-c58a-4ac6-be54-bdfef411fd2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7\" (UID: \"fa84afb2-c58a-4ac6-be54-bdfef411fd2b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7" Oct 30 21:26:13.204515 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:13.204453 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt5bq\" (UniqueName: \"kubernetes.io/projected/fa84afb2-c58a-4ac6-be54-bdfef411fd2b-kube-api-access-mt5bq\") pod \"payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7\" (UID: \"fa84afb2-c58a-4ac6-be54-bdfef411fd2b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7" Oct 30 21:26:13.286747 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:13.286713 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7" Oct 30 21:26:13.703274 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:13.689269 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7"] Oct 30 21:26:14.269536 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:14.269486 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7" event={"ID":"fa84afb2-c58a-4ac6-be54-bdfef411fd2b","Type":"ContainerStarted","Data":"1499fdccf2872bebf8c4621a7f4a5e92498809c53cfc13cc97e299b02d01595e"} Oct 30 21:26:14.270009 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:14.269560 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7" Oct 30 21:26:14.270009 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:14.269586 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7" event={"ID":"fa84afb2-c58a-4ac6-be54-bdfef411fd2b","Type":"ContainerStarted","Data":"7e1db3b97c060a66e32e36f13ead93bcdc77935e95cbe3431e451b746df4a678"} Oct 30 21:26:14.292958 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:14.288127 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7" podStartSLOduration=2.288110226 podStartE2EDuration="2.288110226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:14.287507039 +0000 UTC m=+2879.467403545" watchObservedRunningTime="2026-10-30 21:26:14.288110226 +0000 UTC m=+2879.468006734" Oct 30 21:26:14.776864 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:14.776819 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" containerID="cri-o://11c80ebcee5d12505ed3675e680d3758ae82d311e50d41411fac77ac940a1713" gracePeriod=45 Oct 30 21:26:15.072837 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.072810 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 21:26:15.243208 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.243172 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3111f52b-8837-47a7-9dc2-1da7d0574d1f-kube-api-access-shtp5\" (UniqueName: \"kubernetes.io/projected/3111f52b-8837-47a7-9dc2-1da7d0574d1f-kube-api-access-shtp5\") pod \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " Oct 30 21:26:15.243393 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.243244 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3111f52b-8837-47a7-9dc2-1da7d0574d1f-tls-certs\" (UniqueName: \"kubernetes.io/secret/3111f52b-8837-47a7-9dc2-1da7d0574d1f-tls-certs\") pod \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " Oct 30 21:26:15.243393 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.243326 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-dshm\") pod \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " Oct 30 21:26:15.243529 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.243457 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-home\" (UniqueName: \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-home\") pod \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " Oct 30 21:26:15.243590 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.243529 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-model-cache\") pod \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " Oct 30 21:26:15.244481 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.243692 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-tmp-dir\") pod \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\" (UID: \"3111f52b-8837-47a7-9dc2-1da7d0574d1f\") " Oct 30 21:26:15.244481 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.243811 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-home" pod "3111f52b-8837-47a7-9dc2-1da7d0574d1f" (UID: "3111f52b-8837-47a7-9dc2-1da7d0574d1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:15.244481 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.244071 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-tmp-dir" pod "3111f52b-8837-47a7-9dc2-1da7d0574d1f" (UID: "3111f52b-8837-47a7-9dc2-1da7d0574d1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:15.244481 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.244398 4848 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-home\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:26:15.244481 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.244413 4848 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-tmp-dir\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:26:15.244481 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.244459 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-model-cache" pod "3111f52b-8837-47a7-9dc2-1da7d0574d1f" (UID: "3111f52b-8837-47a7-9dc2-1da7d0574d1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:15.246135 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.246106 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-dshm" pod "3111f52b-8837-47a7-9dc2-1da7d0574d1f" (UID: "3111f52b-8837-47a7-9dc2-1da7d0574d1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:15.247364 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.247339 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3111f52b-8837-47a7-9dc2-1da7d0574d1f-kube-api-access-shtp5" pod "3111f52b-8837-47a7-9dc2-1da7d0574d1f" (UID: "3111f52b-8837-47a7-9dc2-1da7d0574d1f"). InnerVolumeSpecName "kube-api-access-shtp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:15.247508 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.247492 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3111f52b-8837-47a7-9dc2-1da7d0574d1f-tls-certs" pod "3111f52b-8837-47a7-9dc2-1da7d0574d1f" (UID: "3111f52b-8837-47a7-9dc2-1da7d0574d1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:15.280412 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.280380 4848 generic.go:356] "Generic (PLEG): container finished" podID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerID="11c80ebcee5d12505ed3675e680d3758ae82d311e50d41411fac77ac940a1713" exitCode=0 Oct 30 21:26:15.280915 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.280434 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" Oct 30 21:26:15.280915 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.280435 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" event={"ID":"3111f52b-8837-47a7-9dc2-1da7d0574d1f","Type":"ContainerDied","Data":"11c80ebcee5d12505ed3675e680d3758ae82d311e50d41411fac77ac940a1713"} Oct 30 21:26:15.280915 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.280487 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" event={"ID":"3111f52b-8837-47a7-9dc2-1da7d0574d1f","Type":"ContainerDied","Data":"9babff7f575a77a23d2685c901628faeb98478d71752435d4afe90e544b630ac"} Oct 30 21:26:15.280915 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.280502 4848 scope.go:122] "RemoveContainer" containerID="11c80ebcee5d12505ed3675e680d3758ae82d311e50d41411fac77ac940a1713" Oct 30 21:26:15.292855 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.292836 4848 scope.go:122] "RemoveContainer" containerID="11c80ebcee5d12505ed3675e680d3758ae82d311e50d41411fac77ac940a1713" Oct 30 21:26:15.293148 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:26:15.293129 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11c80ebcee5d12505ed3675e680d3758ae82d311e50d41411fac77ac940a1713\": container with ID starting with 11c80ebcee5d12505ed3675e680d3758ae82d311e50d41411fac77ac940a1713 not found: ID does not exist" containerID="11c80ebcee5d12505ed3675e680d3758ae82d311e50d41411fac77ac940a1713" Oct 30 21:26:15.293207 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.293156 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11c80ebcee5d12505ed3675e680d3758ae82d311e50d41411fac77ac940a1713"} err="failed to get container status \"11c80ebcee5d12505ed3675e680d3758ae82d311e50d41411fac77ac940a1713\": rpc error: code = NotFound desc = could not find container \"11c80ebcee5d12505ed3675e680d3758ae82d311e50d41411fac77ac940a1713\": container with ID starting with 11c80ebcee5d12505ed3675e680d3758ae82d311e50d41411fac77ac940a1713 not found: ID does not exist" Oct 30 21:26:15.308808 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.308756 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g"] Oct 30 21:26:15.323890 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.323849 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g"] Oct 30 21:26:15.346218 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.346184 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shtp5\" (UniqueName: \"kubernetes.io/projected/3111f52b-8837-47a7-9dc2-1da7d0574d1f-kube-api-access-shtp5\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:26:15.346218 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.346214 4848 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3111f52b-8837-47a7-9dc2-1da7d0574d1f-tls-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:26:15.346218 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.346225 4848 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-dshm\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:26:15.346456 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.346233 4848 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3111f52b-8837-47a7-9dc2-1da7d0574d1f-model-cache\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:26:15.994751 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:15.994708 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-8e4acb/premium-w2-8e4acb-kserve-7b5b878587-ptp4g" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 21:26:16.936820 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:16.936751 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" path="/var/lib/kubelet/pods/3111f52b-8837-47a7-9dc2-1da7d0574d1f/volumes" Oct 30 21:26:22.512277 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:26:22.512233 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:26:25.283584 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:25.283558 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7" Oct 30 21:26:52.534568 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:26:52.534486 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:26:52.955734 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:52.955699 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz"] Oct 30 21:26:52.955986 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:52.955959 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" podUID="7d33a513-4751-4c10-bbbe-061498a405f3" containerName="maas-api" containerID="cri-o://26888debaa6c97f9fc70b6e5a3b0e018bdcd0dd51401000d5e9c164f6f388625" gracePeriod=30 Oct 30 21:26:53.584141 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.584107 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" Oct 30 21:26:53.631906 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.631286 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d33a513-4751-4c10-bbbe-061498a405f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d33a513-4751-4c10-bbbe-061498a405f3-openshift-service-ca\") pod \"7d33a513-4751-4c10-bbbe-061498a405f3\" (UID: \"7d33a513-4751-4c10-bbbe-061498a405f3\") " Oct 30 21:26:53.635611 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.635586 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d33a513-4751-4c10-bbbe-061498a405f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d33a513-4751-4c10-bbbe-061498a405f3-maas-api-tls\") pod \"7d33a513-4751-4c10-bbbe-061498a405f3\" (UID: \"7d33a513-4751-4c10-bbbe-061498a405f3\") " Oct 30 21:26:53.635950 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.635930 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d33a513-4751-4c10-bbbe-061498a405f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d33a513-4751-4c10-bbbe-061498a405f3-metrics-certs\") pod \"7d33a513-4751-4c10-bbbe-061498a405f3\" (UID: \"7d33a513-4751-4c10-bbbe-061498a405f3\") " Oct 30 21:26:53.636240 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.636221 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d33a513-4751-4c10-bbbe-061498a405f3-kube-api-access-x4nph\" (UniqueName: \"kubernetes.io/projected/7d33a513-4751-4c10-bbbe-061498a405f3-kube-api-access-x4nph\") pod \"7d33a513-4751-4c10-bbbe-061498a405f3\" (UID: \"7d33a513-4751-4c10-bbbe-061498a405f3\") " Oct 30 21:26:53.636966 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.635426 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d33a513-4751-4c10-bbbe-061498a405f3-openshift-service-ca" pod "7d33a513-4751-4c10-bbbe-061498a405f3" (UID: "7d33a513-4751-4c10-bbbe-061498a405f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:53.637334 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.637314 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d33a513-4751-4c10-bbbe-061498a405f3-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:26:53.650557 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.649417 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d33a513-4751-4c10-bbbe-061498a405f3-kube-api-access-x4nph" pod "7d33a513-4751-4c10-bbbe-061498a405f3" (UID: "7d33a513-4751-4c10-bbbe-061498a405f3"). InnerVolumeSpecName "kube-api-access-x4nph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:53.650557 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.650347 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d33a513-4751-4c10-bbbe-061498a405f3-maas-api-tls" pod "7d33a513-4751-4c10-bbbe-061498a405f3" (UID: "7d33a513-4751-4c10-bbbe-061498a405f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:53.650557 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.650431 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d33a513-4751-4c10-bbbe-061498a405f3-metrics-certs" pod "7d33a513-4751-4c10-bbbe-061498a405f3" (UID: "7d33a513-4751-4c10-bbbe-061498a405f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:53.657573 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.657538 4848 generic.go:356] "Generic (PLEG): container finished" podID="7d33a513-4751-4c10-bbbe-061498a405f3" containerID="26888debaa6c97f9fc70b6e5a3b0e018bdcd0dd51401000d5e9c164f6f388625" exitCode=0 Oct 30 21:26:53.657729 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.657710 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" event={"ID":"7d33a513-4751-4c10-bbbe-061498a405f3","Type":"ContainerDied","Data":"26888debaa6c97f9fc70b6e5a3b0e018bdcd0dd51401000d5e9c164f6f388625"} Oct 30 21:26:53.657851 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.657780 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" event={"ID":"7d33a513-4751-4c10-bbbe-061498a405f3","Type":"ContainerDied","Data":"7d0986c290324f9c040c3840194133745072963fe401450157d41a0d9ce8eeba"} Oct 30 21:26:53.657851 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.657825 4848 scope.go:122] "RemoveContainer" containerID="26888debaa6c97f9fc70b6e5a3b0e018bdcd0dd51401000d5e9c164f6f388625" Oct 30 21:26:53.657991 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.657946 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz" Oct 30 21:26:53.682341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.682322 4848 scope.go:122] "RemoveContainer" containerID="26888debaa6c97f9fc70b6e5a3b0e018bdcd0dd51401000d5e9c164f6f388625" Oct 30 21:26:53.682717 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:26:53.682690 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26888debaa6c97f9fc70b6e5a3b0e018bdcd0dd51401000d5e9c164f6f388625\": container with ID starting with 26888debaa6c97f9fc70b6e5a3b0e018bdcd0dd51401000d5e9c164f6f388625 not found: ID does not exist" containerID="26888debaa6c97f9fc70b6e5a3b0e018bdcd0dd51401000d5e9c164f6f388625" Oct 30 21:26:53.682872 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.682726 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26888debaa6c97f9fc70b6e5a3b0e018bdcd0dd51401000d5e9c164f6f388625"} err="failed to get container status \"26888debaa6c97f9fc70b6e5a3b0e018bdcd0dd51401000d5e9c164f6f388625\": rpc error: code = NotFound desc = could not find container \"26888debaa6c97f9fc70b6e5a3b0e018bdcd0dd51401000d5e9c164f6f388625\": container with ID starting with 26888debaa6c97f9fc70b6e5a3b0e018bdcd0dd51401000d5e9c164f6f388625 not found: ID does not exist" Oct 30 21:26:53.690240 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.690216 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz"] Oct 30 21:26:53.694249 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.694220 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-8e4acb-7f56686d64-jxngz"] Oct 30 21:26:53.738539 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.738500 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d33a513-4751-4c10-bbbe-061498a405f3-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:26:53.738539 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.738526 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d33a513-4751-4c10-bbbe-061498a405f3-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:26:53.738539 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:53.738537 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4nph\" (UniqueName: \"kubernetes.io/projected/7d33a513-4751-4c10-bbbe-061498a405f3-kube-api-access-x4nph\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:26:54.936241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:54.936204 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d33a513-4751-4c10-bbbe-061498a405f3" path="/var/lib/kubelet/pods/7d33a513-4751-4c10-bbbe-061498a405f3/volumes" Oct 30 21:26:56.216101 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:56.216056 4848 patch_prober.go:29] interesting pod/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:26:56.216719 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:56.216146 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" podUID="5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:58.023521 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.023490 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r"] Oct 30 21:26:58.025355 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.025333 4848 cpu_manager.go:414] "containerMap: removing container" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" Oct 30 21:26:58.025526 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.025510 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" Oct 30 21:26:58.025695 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.025673 4848 cpu_manager.go:414] "containerMap: removing container" podUID="7d33a513-4751-4c10-bbbe-061498a405f3" containerName="maas-api" Oct 30 21:26:58.025695 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.025693 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d33a513-4751-4c10-bbbe-061498a405f3" containerName="maas-api" Oct 30 21:26:58.026021 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.025910 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d33a513-4751-4c10-bbbe-061498a405f3" containerName="maas-api" Oct 30 21:26:58.026021 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.025939 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="3111f52b-8837-47a7-9dc2-1da7d0574d1f" containerName="main" Oct 30 21:26:58.039674 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.039637 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r"] Oct 30 21:26:58.039851 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.039741 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.045641 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.044327 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f8e2c3-openshift-default-dockercfg-b259x\"" Oct 30 21:26:58.187788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.186494 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-envoy\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.187788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.186573 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-data\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.187788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.186624 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-podinfo\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.187788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.186791 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p659r\" (UniqueName: \"kubernetes.io/projected/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-kube-api-access-p659r\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.187788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.186854 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-workload-socket\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.187788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.186895 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-token\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.187788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.186923 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istiod-ca-cert\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.187788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.186950 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-credential-socket\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.187788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.187019 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-workload-certs\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.356575 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.349444 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p659r\" (UniqueName: \"kubernetes.io/projected/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-kube-api-access-p659r\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.356575 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.349530 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-workload-socket\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.356575 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.349575 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-token\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.356575 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.349606 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istiod-ca-cert\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.356575 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.349640 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-credential-socket\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.356575 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.349701 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-workload-certs\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.356575 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.351164 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-workload-socket\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.356575 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.351453 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-credential-socket\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.356575 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.352175 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istiod-ca-cert\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.356575 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.352474 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-workload-certs\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.359047 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.349935 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-envoy\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.359047 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.358063 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-data\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.359047 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.358115 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-podinfo\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.359047 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.359005 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-data\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.363196 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.363145 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-podinfo\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.366468 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.366421 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-token\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.386115 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.383296 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-envoy\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.387493 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.387441 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p659r\" (UniqueName: \"kubernetes.io/projected/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-kube-api-access-p659r\") pod \"e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.674881 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.674845 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:26:58.961224 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.961193 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r"] Oct 30 21:26:58.961796 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:26:58.961756 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4c39fa_0000_40ad_8f24_53ae45fe0f7e.slice/crio-1592592fe7eaf0ce741fba5f7db124d59ef0d60648b687f331fa9713cf10055e WatchSource:0}: Error finding container 1592592fe7eaf0ce741fba5f7db124d59ef0d60648b687f331fa9713cf10055e: Status 404 returned error can't find the container with id 1592592fe7eaf0ce741fba5f7db124d59ef0d60648b687f331fa9713cf10055e Oct 30 21:26:58.964761 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.964726 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:58.964891 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.964819 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:58.964891 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:58.964851 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:59.709630 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:59.709596 4848 generic.go:356] "Generic (PLEG): container finished" podID="5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" containerID="00f4dc6a43893ba52fa8978eedb79498e9abbf613e28ca9223114d0bed393d67" exitCode=0 Oct 30 21:26:59.710081 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:59.709665 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" event={"ID":"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d","Type":"ContainerDied","Data":"00f4dc6a43893ba52fa8978eedb79498e9abbf613e28ca9223114d0bed393d67"} Oct 30 21:26:59.710165 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:59.710145 4848 scope.go:122] "RemoveContainer" containerID="00f4dc6a43893ba52fa8978eedb79498e9abbf613e28ca9223114d0bed393d67" Oct 30 21:26:59.711525 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:59.711502 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" event={"ID":"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e","Type":"ContainerStarted","Data":"1f17dcfe5a7265ddd079ef9f424a4032a4bf5d488d203b679d6958bc3d3bfe76"} Oct 30 21:26:59.711607 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:59.711546 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" event={"ID":"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e","Type":"ContainerStarted","Data":"1592592fe7eaf0ce741fba5f7db124d59ef0d60648b687f331fa9713cf10055e"} Oct 30 21:26:59.712322 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:59.712292 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:59.712376 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:59.712366 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:59.712415 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:59.712398 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:59.753178 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:26:59.753133 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" podStartSLOduration=2.7531187900000003 podStartE2EDuration="2.75311879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:59.748689792 +0000 UTC m=+2924.928586299" watchObservedRunningTime="2026-10-30 21:26:59.75311879 +0000 UTC m=+2924.933015297" Oct 30 21:27:00.676010 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:00.675970 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:27:00.683551 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:00.683525 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:27:00.720553 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:00.720516 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" event={"ID":"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d","Type":"ContainerStarted","Data":"f211bbb00eb69edfd5e13e120e59d30023b45474ed54186c45e88950c04458c3"} Oct 30 21:27:00.720977 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:00.720682 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:27:00.721520 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:00.721500 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:27:01.302452 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:01.302418 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:27:01.302638 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:01.302592 4848 patch_prober.go:29] interesting pod/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 21:27:01.302690 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:01.302645 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" podUID="5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 21:27:01.856014 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:01.855976 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m"] Oct 30 21:27:01.856653 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:01.856620 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" podUID="cea433a7-4828-4946-9086-3976cea6c973" containerName="istio-proxy" containerID="cri-o://4d853b05930415906f00b32301334f471967b1033e6c61b98ac75098c8ab6a1f" gracePeriod=30 Oct 30 21:27:02.318793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:02.306293 4848 patch_prober.go:29] interesting pod/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 21:27:02.318793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:02.306404 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" podUID="5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 21:27:03.302678 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:03.302641 4848 patch_prober.go:29] interesting pod/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 21:27:03.303125 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:03.302714 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" podUID="5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 21:27:03.727802 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:03.704155 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx"] Oct 30 21:27:03.736860 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:03.736831 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx"] Oct 30 21:27:03.737075 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:03.736944 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx" Oct 30 21:27:03.746200 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:03.746069 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f8e2c3\"" Oct 30 21:27:03.746200 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:03.746157 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f8e2c3-dockercfg-l495n\"" Oct 30 21:27:03.791037 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:03.790998 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr2tm\" (UniqueName: \"kubernetes.io/projected/69dbc4b2-1701-4de6-a810-795807a04c62-kube-api-access-nr2tm\") pod \"payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx\" (UID: \"69dbc4b2-1701-4de6-a810-795807a04c62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx" Oct 30 21:27:03.791037 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:03.791047 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69dbc4b2-1701-4de6-a810-795807a04c62-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx\" (UID: \"69dbc4b2-1701-4de6-a810-795807a04c62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx" Oct 30 21:27:03.934799 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:03.899605 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr2tm\" (UniqueName: \"kubernetes.io/projected/69dbc4b2-1701-4de6-a810-795807a04c62-kube-api-access-nr2tm\") pod \"payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx\" (UID: \"69dbc4b2-1701-4de6-a810-795807a04c62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx" Oct 30 21:27:03.934799 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:03.899661 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69dbc4b2-1701-4de6-a810-795807a04c62-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx\" (UID: \"69dbc4b2-1701-4de6-a810-795807a04c62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx" Oct 30 21:27:03.934799 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:03.900870 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69dbc4b2-1701-4de6-a810-795807a04c62-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx\" (UID: \"69dbc4b2-1701-4de6-a810-795807a04c62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx" Oct 30 21:27:04.034535 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:04.019100 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr2tm\" (UniqueName: \"kubernetes.io/projected/69dbc4b2-1701-4de6-a810-795807a04c62-kube-api-access-nr2tm\") pod \"payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx\" (UID: \"69dbc4b2-1701-4de6-a810-795807a04c62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx" Oct 30 21:27:04.062140 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:04.062102 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx" Oct 30 21:27:04.303080 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:04.302001 4848 patch_prober.go:29] interesting pod/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 21:27:04.303080 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:04.302062 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" podUID="5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 21:27:04.475521 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:04.475492 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx"] Oct 30 21:27:04.476189 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:27:04.476161 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69dbc4b2_1701_4de6_a810_795807a04c62.slice/crio-d0af641cafb21e390bae7cf6c396f422b2c831440857132a3a77a039bcabf315 WatchSource:0}: Error finding container d0af641cafb21e390bae7cf6c396f422b2c831440857132a3a77a039bcabf315: Status 404 returned error can't find the container with id d0af641cafb21e390bae7cf6c396f422b2c831440857132a3a77a039bcabf315 Oct 30 21:27:04.759838 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:04.759799 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx" event={"ID":"69dbc4b2-1701-4de6-a810-795807a04c62","Type":"ContainerStarted","Data":"dc47f4171a9f956d2ee9c563e8c72648c37f29ce9be1a301aee7c2e7299370fa"} Oct 30 21:27:04.759991 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:04.759860 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx" event={"ID":"69dbc4b2-1701-4de6-a810-795807a04c62","Type":"ContainerStarted","Data":"d0af641cafb21e390bae7cf6c396f422b2c831440857132a3a77a039bcabf315"} Oct 30 21:27:04.759991 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:04.759898 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx" Oct 30 21:27:04.776848 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:04.776797 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx" podStartSLOduration=1.776783697 podStartE2EDuration="1.776783697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:04.775199248 +0000 UTC m=+2929.955095748" watchObservedRunningTime="2026-10-30 21:27:04.776783697 +0000 UTC m=+2929.956680200" Oct 30 21:27:05.302056 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:05.302028 4848 patch_prober.go:29] interesting pod/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 21:27:05.302239 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:05.302074 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" podUID="5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 21:27:06.302309 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:06.302271 4848 patch_prober.go:29] interesting pod/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" start-of-body= Oct 30 21:27:06.302726 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:06.302336 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" podUID="5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": dial tcp 10.134.0.238:15021: connect: connection refused" Oct 30 21:27:07.195108 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.195065 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 21:27:07.307701 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.307668 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:27:07.308095 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.308077 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:27:07.308739 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.308720 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:27:07.320224 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.320205 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-credential-socket\") pod \"cea433a7-4828-4946-9086-3976cea6c973\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " Oct 30 21:27:07.320295 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.320253 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea433a7-4828-4946-9086-3976cea6c973-kube-api-access-qgj9n\" (UniqueName: \"kubernetes.io/projected/cea433a7-4828-4946-9086-3976cea6c973-kube-api-access-qgj9n\") pod \"cea433a7-4828-4946-9086-3976cea6c973\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " Oct 30 21:27:07.320295 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.320286 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cea433a7-4828-4946-9086-3976cea6c973-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea433a7-4828-4946-9086-3976cea6c973-istio-podinfo\") pod \"cea433a7-4828-4946-9086-3976cea6c973\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " Oct 30 21:27:07.320385 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.320333 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cea433a7-4828-4946-9086-3976cea6c973-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea433a7-4828-4946-9086-3976cea6c973-istiod-ca-cert\") pod \"cea433a7-4828-4946-9086-3976cea6c973\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " Oct 30 21:27:07.320385 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.320359 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cea433a7-4828-4946-9086-3976cea6c973-istio-token\" (UniqueName: \"kubernetes.io/projected/cea433a7-4828-4946-9086-3976cea6c973-istio-token\") pod \"cea433a7-4828-4946-9086-3976cea6c973\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " Oct 30 21:27:07.320499 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.320488 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-istio-data\") pod \"cea433a7-4828-4946-9086-3976cea6c973\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " Oct 30 21:27:07.320566 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.320507 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-credential-socket" pod "cea433a7-4828-4946-9086-3976cea6c973" (UID: "cea433a7-4828-4946-9086-3976cea6c973"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:07.320624 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.320563 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-istio-envoy\") pod \"cea433a7-4828-4946-9086-3976cea6c973\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " Oct 30 21:27:07.320624 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.320602 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-workload-socket\") pod \"cea433a7-4828-4946-9086-3976cea6c973\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " Oct 30 21:27:07.320743 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.320677 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-workload-certs\") pod \"cea433a7-4828-4946-9086-3976cea6c973\" (UID: \"cea433a7-4828-4946-9086-3976cea6c973\") " Oct 30 21:27:07.320876 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.320825 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-istio-data" pod "cea433a7-4828-4946-9086-3976cea6c973" (UID: "cea433a7-4828-4946-9086-3976cea6c973"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:07.320876 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.320833 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cea433a7-4828-4946-9086-3976cea6c973-istiod-ca-cert" pod "cea433a7-4828-4946-9086-3976cea6c973" (UID: "cea433a7-4828-4946-9086-3976cea6c973"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:27:07.321026 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.321002 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-workload-socket" pod "cea433a7-4828-4946-9086-3976cea6c973" (UID: "cea433a7-4828-4946-9086-3976cea6c973"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:07.321152 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.321129 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-workload-certs" pod "cea433a7-4828-4946-9086-3976cea6c973" (UID: "cea433a7-4828-4946-9086-3976cea6c973"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:07.321467 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.321445 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:27:07.321520 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.321477 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cea433a7-4828-4946-9086-3976cea6c973-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:27:07.321520 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.321493 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:27:07.321520 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.321506 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:27:07.321623 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.321521 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:27:07.322716 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.322694 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cea433a7-4828-4946-9086-3976cea6c973-istio-podinfo" pod "cea433a7-4828-4946-9086-3976cea6c973" (UID: "cea433a7-4828-4946-9086-3976cea6c973"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:27:07.322838 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.322737 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea433a7-4828-4946-9086-3976cea6c973-kube-api-access-qgj9n" pod "cea433a7-4828-4946-9086-3976cea6c973" (UID: "cea433a7-4828-4946-9086-3976cea6c973"). InnerVolumeSpecName "kube-api-access-qgj9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:27:07.323010 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.322995 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-istio-envoy" pod "cea433a7-4828-4946-9086-3976cea6c973" (UID: "cea433a7-4828-4946-9086-3976cea6c973"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:07.323075 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.323055 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cea433a7-4828-4946-9086-3976cea6c973-istio-token" pod "cea433a7-4828-4946-9086-3976cea6c973" (UID: "cea433a7-4828-4946-9086-3976cea6c973"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:27:07.422872 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.422837 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgj9n\" (UniqueName: \"kubernetes.io/projected/cea433a7-4828-4946-9086-3976cea6c973-kube-api-access-qgj9n\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:27:07.422872 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.422872 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cea433a7-4828-4946-9086-3976cea6c973-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:27:07.423198 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.422887 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cea433a7-4828-4946-9086-3976cea6c973-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:27:07.423198 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.422901 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cea433a7-4828-4946-9086-3976cea6c973-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:27:07.785395 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.785360 4848 generic.go:356] "Generic (PLEG): container finished" podID="cea433a7-4828-4946-9086-3976cea6c973" containerID="4d853b05930415906f00b32301334f471967b1033e6c61b98ac75098c8ab6a1f" exitCode=0 Oct 30 21:27:07.785614 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.785408 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" Oct 30 21:27:07.785614 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.785446 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" event={"ID":"cea433a7-4828-4946-9086-3976cea6c973","Type":"ContainerDied","Data":"4d853b05930415906f00b32301334f471967b1033e6c61b98ac75098c8ab6a1f"} Oct 30 21:27:07.785614 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.785530 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m" event={"ID":"cea433a7-4828-4946-9086-3976cea6c973","Type":"ContainerDied","Data":"55d567324e000ab87b69ada9ad9e6d36a0cd25651342a9f245de586d42efa297"} Oct 30 21:27:07.785614 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.785572 4848 scope.go:122] "RemoveContainer" containerID="4d853b05930415906f00b32301334f471967b1033e6c61b98ac75098c8ab6a1f" Oct 30 21:27:07.798224 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.798205 4848 scope.go:122] "RemoveContainer" containerID="4d853b05930415906f00b32301334f471967b1033e6c61b98ac75098c8ab6a1f" Oct 30 21:27:07.798524 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:27:07.798505 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d853b05930415906f00b32301334f471967b1033e6c61b98ac75098c8ab6a1f\": container with ID starting with 4d853b05930415906f00b32301334f471967b1033e6c61b98ac75098c8ab6a1f not found: ID does not exist" containerID="4d853b05930415906f00b32301334f471967b1033e6c61b98ac75098c8ab6a1f" Oct 30 21:27:07.798584 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.798531 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d853b05930415906f00b32301334f471967b1033e6c61b98ac75098c8ab6a1f"} err="failed to get container status \"4d853b05930415906f00b32301334f471967b1033e6c61b98ac75098c8ab6a1f\": rpc error: code = NotFound desc = could not find container \"4d853b05930415906f00b32301334f471967b1033e6c61b98ac75098c8ab6a1f\": container with ID starting with 4d853b05930415906f00b32301334f471967b1033e6c61b98ac75098c8ab6a1f not found: ID does not exist" Oct 30 21:27:07.811142 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.811107 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m"] Oct 30 21:27:07.816059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:07.816035 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-8e4acb-openshift-default-868dc48fff-2qs4m"] Oct 30 21:27:08.934841 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:08.934810 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cea433a7-4828-4946-9086-3976cea6c973" path="/var/lib/kubelet/pods/cea433a7-4828-4946-9086-3976cea6c973/volumes" Oct 30 21:27:15.773560 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:15.773532 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx" Oct 30 21:27:21.878291 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:21.878254 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc"] Oct 30 21:27:21.879387 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:21.879362 4848 cpu_manager.go:414] "containerMap: removing container" podUID="cea433a7-4828-4946-9086-3976cea6c973" containerName="istio-proxy" Oct 30 21:27:21.879601 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:21.879577 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cea433a7-4828-4946-9086-3976cea6c973" containerName="istio-proxy" Oct 30 21:27:21.879908 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:21.879891 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="cea433a7-4828-4946-9086-3976cea6c973" containerName="istio-proxy" Oct 30 21:27:21.885781 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:21.885738 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" Oct 30 21:27:21.900480 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:21.900451 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f8e2c3-serving-cert\"" Oct 30 21:27:21.906784 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:21.905109 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc"] Oct 30 21:27:21.969187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:21.969156 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkvqf\" (UniqueName: \"kubernetes.io/projected/ee6e41be-d07f-4346-a538-8bd375b21b9b-kube-api-access-vkvqf\") pod \"maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc\" (UID: \"ee6e41be-d07f-4346-a538-8bd375b21b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" Oct 30 21:27:21.969336 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:21.969209 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee6e41be-d07f-4346-a538-8bd375b21b9b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc\" (UID: \"ee6e41be-d07f-4346-a538-8bd375b21b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" Oct 30 21:27:21.969396 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:21.969342 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee6e41be-d07f-4346-a538-8bd375b21b9b-maas-api-tls\") pod \"maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc\" (UID: \"ee6e41be-d07f-4346-a538-8bd375b21b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" Oct 30 21:27:21.969464 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:21.969445 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee6e41be-d07f-4346-a538-8bd375b21b9b-metrics-certs\") pod \"maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc\" (UID: \"ee6e41be-d07f-4346-a538-8bd375b21b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" Oct 30 21:27:22.071321 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:22.071289 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkvqf\" (UniqueName: \"kubernetes.io/projected/ee6e41be-d07f-4346-a538-8bd375b21b9b-kube-api-access-vkvqf\") pod \"maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc\" (UID: \"ee6e41be-d07f-4346-a538-8bd375b21b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" Oct 30 21:27:22.071523 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:22.071347 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee6e41be-d07f-4346-a538-8bd375b21b9b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc\" (UID: \"ee6e41be-d07f-4346-a538-8bd375b21b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" Oct 30 21:27:22.071523 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:22.071417 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee6e41be-d07f-4346-a538-8bd375b21b9b-maas-api-tls\") pod \"maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc\" (UID: \"ee6e41be-d07f-4346-a538-8bd375b21b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" Oct 30 21:27:22.071523 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:22.071487 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee6e41be-d07f-4346-a538-8bd375b21b9b-metrics-certs\") pod \"maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc\" (UID: \"ee6e41be-d07f-4346-a538-8bd375b21b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" Oct 30 21:27:22.072107 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:22.072080 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee6e41be-d07f-4346-a538-8bd375b21b9b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc\" (UID: \"ee6e41be-d07f-4346-a538-8bd375b21b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" Oct 30 21:27:22.074006 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:22.073987 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee6e41be-d07f-4346-a538-8bd375b21b9b-maas-api-tls\") pod \"maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc\" (UID: \"ee6e41be-d07f-4346-a538-8bd375b21b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" Oct 30 21:27:22.074109 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:22.074050 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee6e41be-d07f-4346-a538-8bd375b21b9b-metrics-certs\") pod \"maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc\" (UID: \"ee6e41be-d07f-4346-a538-8bd375b21b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" Oct 30 21:27:22.082066 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:22.082044 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkvqf\" (UniqueName: \"kubernetes.io/projected/ee6e41be-d07f-4346-a538-8bd375b21b9b-kube-api-access-vkvqf\") pod \"maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc\" (UID: \"ee6e41be-d07f-4346-a538-8bd375b21b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" Oct 30 21:27:22.212195 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:22.212170 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" Oct 30 21:27:22.496748 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:27:22.496708 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee6e41be_d07f_4346_a538_8bd375b21b9b.slice/crio-200c4e145cc2eed05ed03f6e6e1501a8e2412efa847367e4f25640f65cfa45f7 WatchSource:0}: Error finding container 200c4e145cc2eed05ed03f6e6e1501a8e2412efa847367e4f25640f65cfa45f7: Status 404 returned error can't find the container with id 200c4e145cc2eed05ed03f6e6e1501a8e2412efa847367e4f25640f65cfa45f7 Oct 30 21:27:22.496929 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:22.496883 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc"] Oct 30 21:27:22.517074 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:27:22.517041 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:27:22.940565 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:22.940535 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" event={"ID":"ee6e41be-d07f-4346-a538-8bd375b21b9b","Type":"ContainerStarted","Data":"200c4e145cc2eed05ed03f6e6e1501a8e2412efa847367e4f25640f65cfa45f7"} Oct 30 21:27:23.950207 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:23.950017 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" event={"ID":"ee6e41be-d07f-4346-a538-8bd375b21b9b","Type":"ContainerStarted","Data":"0d065528b126fa7258fdd06d699390fac017475bfc83ba41ee54ef8ece493bee"} Oct 30 21:27:23.950207 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:23.950076 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" Oct 30 21:27:23.969440 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:23.969394 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" podStartSLOduration=2.547990942 podStartE2EDuration="2.969380588s" totalImagesPullingTime="421.389646ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:23.965426392 +0000 UTC m=+2949.145322899" watchObservedRunningTime="2026-10-30 21:27:23.969380588 +0000 UTC m=+2949.149277106" Oct 30 21:27:29.962480 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:29.962446 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" Oct 30 21:27:52.516414 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:27:52.516373 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:27:55.341200 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:55.341167 4848 generic.go:356] "Generic (PLEG): container finished" podID="4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" containerID="1f17dcfe5a7265ddd079ef9f424a4032a4bf5d488d203b679d6958bc3d3bfe76" exitCode=0 Oct 30 21:27:55.341650 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:55.341220 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" event={"ID":"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e","Type":"ContainerDied","Data":"1f17dcfe5a7265ddd079ef9f424a4032a4bf5d488d203b679d6958bc3d3bfe76"} Oct 30 21:27:55.341702 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:55.341660 4848 scope.go:122] "RemoveContainer" containerID="1f17dcfe5a7265ddd079ef9f424a4032a4bf5d488d203b679d6958bc3d3bfe76" Oct 30 21:27:55.343433 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:55.343405 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:27:55.343516 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:55.343472 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:27:55.343516 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:55.343501 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:27:56.351261 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:56.351229 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" event={"ID":"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e","Type":"ContainerStarted","Data":"0fbc51e19561e98c35558c4d36169bb7cb5674d2695abe00c7f2fb69668b08e9"} Oct 30 21:27:56.675826 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:56.675792 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:27:56.676961 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:56.676934 4848 patch_prober.go:29] interesting pod/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 30 21:27:56.677122 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:56.676984 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" podUID="4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 30 21:27:57.675174 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:57.675136 4848 patch_prober.go:29] interesting pod/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 30 21:27:57.675583 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:57.675193 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" podUID="4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 30 21:27:58.675572 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:58.675534 4848 patch_prober.go:29] interesting pod/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 30 21:27:58.676099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:58.675595 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" podUID="4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 30 21:27:59.678065 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:59.678025 4848 patch_prober.go:29] interesting pod/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 30 21:27:59.678661 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:27:59.678629 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" podUID="4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 30 21:28:00.675806 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:00.675751 4848 patch_prober.go:29] interesting pod/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 30 21:28:00.675987 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:00.675844 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" podUID="4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 30 21:28:00.729816 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:00.724860 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:28:01.675781 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:01.675736 4848 patch_prober.go:29] interesting pod/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Oct 30 21:28:01.675971 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:01.675807 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" podUID="4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Oct 30 21:28:02.681836 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:02.681731 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:28:02.683713 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:02.683684 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:28:16.562315 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:16.562236 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 21:28:16.568441 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:16.568419 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 21:28:22.517199 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:28:22.517162 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:28:35.775570 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:35.775535 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84"] Oct 30 21:28:35.776226 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:35.775804 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" podUID="8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" containerName="istio-proxy" containerID="cri-o://7bdf73f56e3d4e0a2c68d166ed6927dd0a13e73b9a3ab980fd678cf9c576b753" gracePeriod=30 Oct 30 21:28:41.061099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.061027 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 21:28:41.104024 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.103988 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-data\") pod \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " Oct 30 21:28:41.104235 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.104074 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-credential-socket\") pod \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " Oct 30 21:28:41.104235 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.104131 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-workload-socket\") pod \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " Oct 30 21:28:41.104235 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.104190 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-envoy\") pod \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " Oct 30 21:28:41.104455 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.104267 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-token\" (UniqueName: \"kubernetes.io/projected/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-token\") pod \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " Oct 30 21:28:41.104455 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.104306 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-podinfo\") pod \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " Oct 30 21:28:41.104455 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.104304 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-data" pod "8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" (UID: "8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:41.104455 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.104420 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-workload-certs\") pod \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " Oct 30 21:28:41.104682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.104463 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-workload-socket" pod "8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" (UID: "8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:41.104682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.104501 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istiod-ca-cert\") pod \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " Oct 30 21:28:41.104682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.104550 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-kube-api-access-g22g4\" (UniqueName: \"kubernetes.io/projected/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-kube-api-access-g22g4\") pod \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\" (UID: \"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4\") " Oct 30 21:28:41.104682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.104604 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-credential-socket" pod "8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" (UID: "8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:41.105300 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.105272 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.105479 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.105303 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.105479 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.105321 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.106118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.106036 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-workload-certs" pod "8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" (UID: "8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:41.106506 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.106465 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istiod-ca-cert" pod "8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" (UID: "8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:28:41.107643 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.107617 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-podinfo" pod "8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" (UID: "8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:28:41.108166 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.108128 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-kube-api-access-g22g4" pod "8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" (UID: "8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4"). InnerVolumeSpecName "kube-api-access-g22g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:28:41.108278 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.108207 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-envoy" pod "8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" (UID: "8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:41.110331 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.110303 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-token" pod "8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" (UID: "8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:28:41.206943 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.206912 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.206943 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.206940 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.206943 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.206950 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g22g4\" (UniqueName: \"kubernetes.io/projected/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-kube-api-access-g22g4\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.207201 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.206959 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.207201 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.206972 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.207201 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.206982 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:28:41.878000 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.877966 4848 generic.go:356] "Generic (PLEG): container finished" podID="8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" containerID="7bdf73f56e3d4e0a2c68d166ed6927dd0a13e73b9a3ab980fd678cf9c576b753" exitCode=0 Oct 30 21:28:41.878188 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.878021 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" Oct 30 21:28:41.878188 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.878045 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" event={"ID":"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4","Type":"ContainerDied","Data":"7bdf73f56e3d4e0a2c68d166ed6927dd0a13e73b9a3ab980fd678cf9c576b753"} Oct 30 21:28:41.878188 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.878105 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84" event={"ID":"8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4","Type":"ContainerDied","Data":"477703af9b146492f0a38d19489b96b07d0702e0803c4748efa615677c2eb603"} Oct 30 21:28:41.878188 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.878137 4848 scope.go:122] "RemoveContainer" containerID="7bdf73f56e3d4e0a2c68d166ed6927dd0a13e73b9a3ab980fd678cf9c576b753" Oct 30 21:28:41.890261 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.890238 4848 scope.go:122] "RemoveContainer" containerID="fa2a27fc7a0dc4cf57ca23ba474f5227396d6e7d6ad364b24dd0edbbc205e11f" Oct 30 21:28:41.903561 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.903350 4848 scope.go:122] "RemoveContainer" containerID="7bdf73f56e3d4e0a2c68d166ed6927dd0a13e73b9a3ab980fd678cf9c576b753" Oct 30 21:28:41.903824 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:28:41.903756 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bdf73f56e3d4e0a2c68d166ed6927dd0a13e73b9a3ab980fd678cf9c576b753\": container with ID starting with 7bdf73f56e3d4e0a2c68d166ed6927dd0a13e73b9a3ab980fd678cf9c576b753 not found: ID does not exist" containerID="7bdf73f56e3d4e0a2c68d166ed6927dd0a13e73b9a3ab980fd678cf9c576b753" Oct 30 21:28:41.903914 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.903839 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bdf73f56e3d4e0a2c68d166ed6927dd0a13e73b9a3ab980fd678cf9c576b753"} err="failed to get container status \"7bdf73f56e3d4e0a2c68d166ed6927dd0a13e73b9a3ab980fd678cf9c576b753\": rpc error: code = NotFound desc = could not find container \"7bdf73f56e3d4e0a2c68d166ed6927dd0a13e73b9a3ab980fd678cf9c576b753\": container with ID starting with 7bdf73f56e3d4e0a2c68d166ed6927dd0a13e73b9a3ab980fd678cf9c576b753 not found: ID does not exist" Oct 30 21:28:41.903914 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.903859 4848 scope.go:122] "RemoveContainer" containerID="fa2a27fc7a0dc4cf57ca23ba474f5227396d6e7d6ad364b24dd0edbbc205e11f" Oct 30 21:28:41.904185 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:28:41.904165 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2a27fc7a0dc4cf57ca23ba474f5227396d6e7d6ad364b24dd0edbbc205e11f\": container with ID starting with fa2a27fc7a0dc4cf57ca23ba474f5227396d6e7d6ad364b24dd0edbbc205e11f not found: ID does not exist" containerID="fa2a27fc7a0dc4cf57ca23ba474f5227396d6e7d6ad364b24dd0edbbc205e11f" Oct 30 21:28:41.904246 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.904195 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2a27fc7a0dc4cf57ca23ba474f5227396d6e7d6ad364b24dd0edbbc205e11f"} err="failed to get container status \"fa2a27fc7a0dc4cf57ca23ba474f5227396d6e7d6ad364b24dd0edbbc205e11f\": rpc error: code = NotFound desc = could not find container \"fa2a27fc7a0dc4cf57ca23ba474f5227396d6e7d6ad364b24dd0edbbc205e11f\": container with ID starting with fa2a27fc7a0dc4cf57ca23ba474f5227396d6e7d6ad364b24dd0edbbc205e11f not found: ID does not exist" Oct 30 21:28:41.904758 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.904743 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84"] Oct 30 21:28:41.913717 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:41.912118 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-f9a819-openshift-default-78c5969f59-g9f84"] Oct 30 21:28:42.671782 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:42.671735 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg"] Oct 30 21:28:42.672221 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:42.672020 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" podUID="91bae7a3-eafb-4ec3-976b-d885576ecca1" containerName="istio-proxy" containerID="cri-o://24a534dbf554343df6b7fe2945cf6d7273c268a3a4e4887c7db28cd6f23fdd18" gracePeriod=30 Oct 30 21:28:42.936016 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:42.935981 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" path="/var/lib/kubelet/pods/8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4/volumes" Oct 30 21:28:47.957810 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:47.957703 4848 generic.go:356] "Generic (PLEG): container finished" podID="91bae7a3-eafb-4ec3-976b-d885576ecca1" containerID="24a534dbf554343df6b7fe2945cf6d7273c268a3a4e4887c7db28cd6f23fdd18" exitCode=0 Oct 30 21:28:47.958238 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:47.957972 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" event={"ID":"91bae7a3-eafb-4ec3-976b-d885576ecca1","Type":"ContainerDied","Data":"24a534dbf554343df6b7fe2945cf6d7273c268a3a4e4887c7db28cd6f23fdd18"} Oct 30 21:28:47.958238 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:47.958031 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" event={"ID":"91bae7a3-eafb-4ec3-976b-d885576ecca1","Type":"ContainerDied","Data":"525152342a242a7807905de8efeffc5df8d777d1096a93e6028aab448dd1381a"} Oct 30 21:28:47.958238 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:47.958058 4848 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="525152342a242a7807905de8efeffc5df8d777d1096a93e6028aab448dd1381a" Oct 30 21:28:47.962615 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:47.962594 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:28:48.074595 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.074561 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91bae7a3-eafb-4ec3-976b-d885576ecca1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91bae7a3-eafb-4ec3-976b-d885576ecca1-istiod-ca-cert\") pod \"91bae7a3-eafb-4ec3-976b-d885576ecca1\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " Oct 30 21:28:48.074810 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.074680 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-credential-socket\") pod \"91bae7a3-eafb-4ec3-976b-d885576ecca1\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " Oct 30 21:28:48.074810 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.074752 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-podinfo\") pod \"91bae7a3-eafb-4ec3-976b-d885576ecca1\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " Oct 30 21:28:48.074958 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.074857 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-workload-certs\") pod \"91bae7a3-eafb-4ec3-976b-d885576ecca1\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " Oct 30 21:28:48.074958 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.074892 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-envoy\") pod \"91bae7a3-eafb-4ec3-976b-d885576ecca1\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " Oct 30 21:28:48.074958 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.074936 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-token\" (UniqueName: \"kubernetes.io/projected/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-token\") pod \"91bae7a3-eafb-4ec3-976b-d885576ecca1\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " Oct 30 21:28:48.075118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.074970 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-workload-socket\") pod \"91bae7a3-eafb-4ec3-976b-d885576ecca1\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " Oct 30 21:28:48.075118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.074979 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-credential-socket" pod "91bae7a3-eafb-4ec3-976b-d885576ecca1" (UID: "91bae7a3-eafb-4ec3-976b-d885576ecca1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:48.075118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.075014 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-data\") pod \"91bae7a3-eafb-4ec3-976b-d885576ecca1\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " Oct 30 21:28:48.075118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.075056 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91bae7a3-eafb-4ec3-976b-d885576ecca1-istiod-ca-cert" pod "91bae7a3-eafb-4ec3-976b-d885576ecca1" (UID: "91bae7a3-eafb-4ec3-976b-d885576ecca1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:28:48.075118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.075102 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91bae7a3-eafb-4ec3-976b-d885576ecca1-kube-api-access-7vn2z\" (UniqueName: \"kubernetes.io/projected/91bae7a3-eafb-4ec3-976b-d885576ecca1-kube-api-access-7vn2z\") pod \"91bae7a3-eafb-4ec3-976b-d885576ecca1\" (UID: \"91bae7a3-eafb-4ec3-976b-d885576ecca1\") " Oct 30 21:28:48.075367 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.075209 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-workload-socket" pod "91bae7a3-eafb-4ec3-976b-d885576ecca1" (UID: "91bae7a3-eafb-4ec3-976b-d885576ecca1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:48.075367 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.075229 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-workload-certs" pod "91bae7a3-eafb-4ec3-976b-d885576ecca1" (UID: "91bae7a3-eafb-4ec3-976b-d885576ecca1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:48.075456 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.075395 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-data" pod "91bae7a3-eafb-4ec3-976b-d885576ecca1" (UID: "91bae7a3-eafb-4ec3-976b-d885576ecca1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:48.075925 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.075896 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:28:48.076040 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.075927 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:28:48.076040 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.075944 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91bae7a3-eafb-4ec3-976b-d885576ecca1-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:28:48.076040 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.075960 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:28:48.076040 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.075973 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:28:48.077271 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.077243 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-envoy" pod "91bae7a3-eafb-4ec3-976b-d885576ecca1" (UID: "91bae7a3-eafb-4ec3-976b-d885576ecca1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:28:48.077271 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.077243 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-podinfo" pod "91bae7a3-eafb-4ec3-976b-d885576ecca1" (UID: "91bae7a3-eafb-4ec3-976b-d885576ecca1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:28:48.077403 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.077290 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-token" pod "91bae7a3-eafb-4ec3-976b-d885576ecca1" (UID: "91bae7a3-eafb-4ec3-976b-d885576ecca1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:28:48.077497 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.077479 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91bae7a3-eafb-4ec3-976b-d885576ecca1-kube-api-access-7vn2z" pod "91bae7a3-eafb-4ec3-976b-d885576ecca1" (UID: "91bae7a3-eafb-4ec3-976b-d885576ecca1"). InnerVolumeSpecName "kube-api-access-7vn2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:28:48.177227 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.177192 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:28:48.177227 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.177221 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:28:48.177227 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.177231 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91bae7a3-eafb-4ec3-976b-d885576ecca1-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:28:48.177484 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.177241 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vn2z\" (UniqueName: \"kubernetes.io/projected/91bae7a3-eafb-4ec3-976b-d885576ecca1-kube-api-access-7vn2z\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:28:48.964883 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.964853 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg" Oct 30 21:28:48.985654 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.985622 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg"] Oct 30 21:28:48.991375 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:48.991350 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-mnctg"] Oct 30 21:28:50.934662 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:28:50.934629 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91bae7a3-eafb-4ec3-976b-d885576ecca1" path="/var/lib/kubelet/pods/91bae7a3-eafb-4ec3-976b-d885576ecca1/volumes" Oct 30 21:28:52.557438 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:28:52.557397 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:29:09.522077 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.522040 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv"] Oct 30 21:29:09.522755 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.522740 4848 cpu_manager.go:414] "containerMap: removing container" podUID="8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" containerName="istio-proxy" Oct 30 21:29:09.522844 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.522755 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" containerName="istio-proxy" Oct 30 21:29:09.522844 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.522799 4848 cpu_manager.go:414] "containerMap: removing container" podUID="8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" containerName="istio-proxy" Oct 30 21:29:09.522844 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.522806 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" containerName="istio-proxy" Oct 30 21:29:09.522844 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.522831 4848 cpu_manager.go:414] "containerMap: removing container" podUID="91bae7a3-eafb-4ec3-976b-d885576ecca1" containerName="istio-proxy" Oct 30 21:29:09.522844 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.522837 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91bae7a3-eafb-4ec3-976b-d885576ecca1" containerName="istio-proxy" Oct 30 21:29:09.523015 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.522936 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" containerName="istio-proxy" Oct 30 21:29:09.523015 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.522947 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="91bae7a3-eafb-4ec3-976b-d885576ecca1" containerName="istio-proxy" Oct 30 21:29:09.526967 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.526944 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.530886 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.530863 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5506f4-openshift-default-dockercfg-jjrlt\"" Oct 30 21:29:09.539279 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.539250 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv"] Oct 30 21:29:09.573730 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.573682 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-token\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.573993 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.573858 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2smmh\" (UniqueName: \"kubernetes.io/projected/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-kube-api-access-2smmh\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.574059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.574004 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istiod-ca-cert\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.574118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.574054 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-workload-certs\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.574118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.574080 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-envoy\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.574282 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.574258 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-credential-socket\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.574325 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.574294 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-podinfo\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.574414 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.574393 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-data\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.574472 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.574454 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-workload-socket\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.675801 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.675725 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-credential-socket\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.675801 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.675784 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-podinfo\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.676078 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.675824 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-data\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.676078 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.675862 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-workload-socket\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.676078 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.675897 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-token\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.676078 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.675982 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2smmh\" (UniqueName: \"kubernetes.io/projected/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-kube-api-access-2smmh\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.676324 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.676088 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istiod-ca-cert\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.676324 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.676130 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-workload-certs\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.676324 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.676154 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-envoy\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.676324 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.676242 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-credential-socket\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.676324 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.676280 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-data\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.676626 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.676333 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-workload-socket\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.676626 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.676568 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-workload-certs\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.676974 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.676949 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istiod-ca-cert\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.678431 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.678402 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-envoy\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.678527 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.678514 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-podinfo\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.685026 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.684996 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-token\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.685170 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.685152 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2smmh\" (UniqueName: \"kubernetes.io/projected/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-kube-api-access-2smmh\") pod \"e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:09.842190 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:09.842091 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:10.153199 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:10.153170 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv"] Oct 30 21:29:10.153881 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:29:10.153851 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31751bb_34fe_4e1f_8bcf_2f9b457d3379.slice/crio-79abc38be52276b7b78ea55587accc5898eba1255a834228c8988dd7d8505664 WatchSource:0}: Error finding container 79abc38be52276b7b78ea55587accc5898eba1255a834228c8988dd7d8505664: Status 404 returned error can't find the container with id 79abc38be52276b7b78ea55587accc5898eba1255a834228c8988dd7d8505664 Oct 30 21:29:10.157317 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:10.157280 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:29:10.157411 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:10.157353 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:29:10.157411 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:10.157385 4848 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:29:10.257992 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:10.257937 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" event={"ID":"c31751bb-34fe-4e1f-8bcf-2f9b457d3379","Type":"ContainerStarted","Data":"d92f44dd44cd4143c49b99d280d55b31616f254101690807bd13b10a878f58d2"} Oct 30 21:29:10.258189 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:10.258011 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" event={"ID":"c31751bb-34fe-4e1f-8bcf-2f9b457d3379","Type":"ContainerStarted","Data":"79abc38be52276b7b78ea55587accc5898eba1255a834228c8988dd7d8505664"} Oct 30 21:29:10.277613 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:10.277554 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" podStartSLOduration=1.277536155 podStartE2EDuration="1.277536155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:10.276099898 +0000 UTC m=+3055.455996405" watchObservedRunningTime="2026-10-30 21:29:10.277536155 +0000 UTC m=+3055.457432664" Oct 30 21:29:10.842890 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:10.842852 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:11.851938 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:11.851893 4848 patch_prober.go:29] interesting pod/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 30 21:29:11.853601 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:11.853563 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" podUID="c31751bb-34fe-4e1f-8bcf-2f9b457d3379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 30 21:29:12.843038 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:12.842996 4848 patch_prober.go:29] interesting pod/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 30 21:29:12.843225 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:12.843054 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" podUID="c31751bb-34fe-4e1f-8bcf-2f9b457d3379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 30 21:29:13.843400 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:13.843361 4848 patch_prober.go:29] interesting pod/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 30 21:29:13.843834 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:13.843429 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" podUID="c31751bb-34fe-4e1f-8bcf-2f9b457d3379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 30 21:29:14.843277 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:14.843242 4848 patch_prober.go:29] interesting pod/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 30 21:29:14.843483 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:14.843305 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" podUID="c31751bb-34fe-4e1f-8bcf-2f9b457d3379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 30 21:29:15.842970 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:15.842936 4848 patch_prober.go:29] interesting pod/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 30 21:29:15.843133 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:15.842990 4848 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" podUID="c31751bb-34fe-4e1f-8bcf-2f9b457d3379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 30 21:29:16.848047 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:16.848012 4848 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:16.848555 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:16.848396 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:16.849182 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:16.849159 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:29:18.016882 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:18.016846 4848 scope.go:122] "RemoveContainer" containerID="24a534dbf554343df6b7fe2945cf6d7273c268a3a4e4887c7db28cd6f23fdd18" Oct 30 21:29:22.512413 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:29:22.512368 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:29:44.308024 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:44.307820 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6"] Oct 30 21:29:46.137646 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:46.137613 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx"] Oct 30 21:29:46.138099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:46.137867 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" podUID="4cfb5025-0b7e-4697-bd4e-e65fff80e66c" containerName="istio-proxy" containerID="cri-o://3932125bafc95caf06c283283e6de914f9eadc6ad43785be1145c4931decb0de" gracePeriod=30 Oct 30 21:29:49.086537 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:49.086500 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:29:49.086983 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:49.086571 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" podUID="4cfb5025-0b7e-4697-bd4e-e65fff80e66c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:29:51.447238 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.447196 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:29:51.519958 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.519747 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-data\") pod \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " Oct 30 21:29:51.519958 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.519866 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istiod-ca-cert\") pod \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " Oct 30 21:29:51.519958 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.519952 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-token\" (UniqueName: \"kubernetes.io/projected/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-token\") pod \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " Oct 30 21:29:51.520297 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.520068 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-podinfo\") pod \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " Oct 30 21:29:51.520297 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.520188 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-workload-socket\") pod \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " Oct 30 21:29:51.520297 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.520248 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-credential-socket\") pod \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " Oct 30 21:29:51.520486 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.520355 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-envoy\") pod \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " Oct 30 21:29:51.520486 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.520412 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-workload-certs\") pod \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " Oct 30 21:29:51.520486 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.520448 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-kube-api-access-ksprd\" (UniqueName: \"kubernetes.io/projected/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-kube-api-access-ksprd\") pod \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\" (UID: \"4cfb5025-0b7e-4697-bd4e-e65fff80e66c\") " Oct 30 21:29:51.521921 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.521883 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-data" pod "4cfb5025-0b7e-4697-bd4e-e65fff80e66c" (UID: "4cfb5025-0b7e-4697-bd4e-e65fff80e66c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:51.522171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.522142 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-workload-socket" pod "4cfb5025-0b7e-4697-bd4e-e65fff80e66c" (UID: "4cfb5025-0b7e-4697-bd4e-e65fff80e66c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:51.524731 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.524665 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-workload-certs" pod "4cfb5025-0b7e-4697-bd4e-e65fff80e66c" (UID: "4cfb5025-0b7e-4697-bd4e-e65fff80e66c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:51.525094 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.525055 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istiod-ca-cert" pod "4cfb5025-0b7e-4697-bd4e-e65fff80e66c" (UID: "4cfb5025-0b7e-4697-bd4e-e65fff80e66c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:51.525266 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.525233 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-credential-socket" pod "4cfb5025-0b7e-4697-bd4e-e65fff80e66c" (UID: "4cfb5025-0b7e-4697-bd4e-e65fff80e66c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:51.525442 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.525417 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-token" pod "4cfb5025-0b7e-4697-bd4e-e65fff80e66c" (UID: "4cfb5025-0b7e-4697-bd4e-e65fff80e66c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:51.525944 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.525917 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-podinfo" pod "4cfb5025-0b7e-4697-bd4e-e65fff80e66c" (UID: "4cfb5025-0b7e-4697-bd4e-e65fff80e66c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:29:51.526344 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.526312 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-kube-api-access-ksprd" pod "4cfb5025-0b7e-4697-bd4e-e65fff80e66c" (UID: "4cfb5025-0b7e-4697-bd4e-e65fff80e66c"). InnerVolumeSpecName "kube-api-access-ksprd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:51.529028 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.528996 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-envoy" pod "4cfb5025-0b7e-4697-bd4e-e65fff80e66c" (UID: "4cfb5025-0b7e-4697-bd4e-e65fff80e66c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:51.622678 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.622634 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:51.622678 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.622670 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:51.622678 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.622680 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksprd\" (UniqueName: \"kubernetes.io/projected/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-kube-api-access-ksprd\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:51.622678 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.622689 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:51.623022 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.622698 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:51.623022 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.622710 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:51.623022 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.622718 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:51.623022 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.622727 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:51.623022 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.622735 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cfb5025-0b7e-4697-bd4e-e65fff80e66c-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:51.667172 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.667083 4848 generic.go:356] "Generic (PLEG): container finished" podID="4cfb5025-0b7e-4697-bd4e-e65fff80e66c" containerID="3932125bafc95caf06c283283e6de914f9eadc6ad43785be1145c4931decb0de" exitCode=0 Oct 30 21:29:51.667172 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.667125 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" event={"ID":"4cfb5025-0b7e-4697-bd4e-e65fff80e66c","Type":"ContainerDied","Data":"3932125bafc95caf06c283283e6de914f9eadc6ad43785be1145c4931decb0de"} Oct 30 21:29:51.667172 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.667150 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" Oct 30 21:29:51.667442 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.667197 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx" event={"ID":"4cfb5025-0b7e-4697-bd4e-e65fff80e66c","Type":"ContainerDied","Data":"8da43e92c868dcda8477f9f7dd721513d11a4110430346db6e27c7a265956b52"} Oct 30 21:29:51.667442 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.667229 4848 scope.go:122] "RemoveContainer" containerID="3932125bafc95caf06c283283e6de914f9eadc6ad43785be1145c4931decb0de" Oct 30 21:29:51.679340 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.679320 4848 scope.go:122] "RemoveContainer" containerID="a953cf9b73616a9d21a449287deb844632e45d3f9766946e80557897406ade79" Oct 30 21:29:51.733226 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.726812 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx"] Oct 30 21:29:51.733226 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.728621 4848 scope.go:122] "RemoveContainer" containerID="3932125bafc95caf06c283283e6de914f9eadc6ad43785be1145c4931decb0de" Oct 30 21:29:51.733226 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:29:51.730472 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3932125bafc95caf06c283283e6de914f9eadc6ad43785be1145c4931decb0de\": container with ID starting with 3932125bafc95caf06c283283e6de914f9eadc6ad43785be1145c4931decb0de not found: ID does not exist" containerID="3932125bafc95caf06c283283e6de914f9eadc6ad43785be1145c4931decb0de" Oct 30 21:29:51.733226 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.730510 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3932125bafc95caf06c283283e6de914f9eadc6ad43785be1145c4931decb0de"} err="failed to get container status \"3932125bafc95caf06c283283e6de914f9eadc6ad43785be1145c4931decb0de\": rpc error: code = NotFound desc = could not find container \"3932125bafc95caf06c283283e6de914f9eadc6ad43785be1145c4931decb0de\": container with ID starting with 3932125bafc95caf06c283283e6de914f9eadc6ad43785be1145c4931decb0de not found: ID does not exist" Oct 30 21:29:51.733226 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.730536 4848 scope.go:122] "RemoveContainer" containerID="a953cf9b73616a9d21a449287deb844632e45d3f9766946e80557897406ade79" Oct 30 21:29:51.733226 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:29:51.730910 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a953cf9b73616a9d21a449287deb844632e45d3f9766946e80557897406ade79\": container with ID starting with a953cf9b73616a9d21a449287deb844632e45d3f9766946e80557897406ade79 not found: ID does not exist" containerID="a953cf9b73616a9d21a449287deb844632e45d3f9766946e80557897406ade79" Oct 30 21:29:51.733226 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.730943 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a953cf9b73616a9d21a449287deb844632e45d3f9766946e80557897406ade79"} err="failed to get container status \"a953cf9b73616a9d21a449287deb844632e45d3f9766946e80557897406ade79\": rpc error: code = NotFound desc = could not find container \"a953cf9b73616a9d21a449287deb844632e45d3f9766946e80557897406ade79\": container with ID starting with a953cf9b73616a9d21a449287deb844632e45d3f9766946e80557897406ade79 not found: ID does not exist" Oct 30 21:29:51.740859 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:51.740811 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-00909f-openshift-default-7c565998f7-mgpgx"] Oct 30 21:29:52.559802 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:29:52.559750 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:29:52.934271 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:52.934238 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cfb5025-0b7e-4697-bd4e-e65fff80e66c" path="/var/lib/kubelet/pods/4cfb5025-0b7e-4697-bd4e-e65fff80e66c/volumes" Oct 30 21:29:55.332480 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:55.332388 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x"] Oct 30 21:29:59.335910 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.335866 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" podUID="24daadef-9dca-41c5-876f-9210ce533955" containerName="main" containerID="cri-o://fe07862ce66ec71411925c6d6b725e76a045527023f09cb8b07aeab769e23ca8" gracePeriod=45 Oct 30 21:29:59.632860 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.632823 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:29:59.765849 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.765817 4848 generic.go:356] "Generic (PLEG): container finished" podID="24daadef-9dca-41c5-876f-9210ce533955" containerID="fe07862ce66ec71411925c6d6b725e76a045527023f09cb8b07aeab769e23ca8" exitCode=0 Oct 30 21:29:59.766058 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.765867 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" Oct 30 21:29:59.766058 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.765910 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" event={"ID":"24daadef-9dca-41c5-876f-9210ce533955","Type":"ContainerDied","Data":"fe07862ce66ec71411925c6d6b725e76a045527023f09cb8b07aeab769e23ca8"} Oct 30 21:29:59.766058 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.765960 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6" event={"ID":"24daadef-9dca-41c5-876f-9210ce533955","Type":"ContainerDied","Data":"a106395c24daa501e0d91b6c03eca8ed4f3adc747905663af4926472059ef9bf"} Oct 30 21:29:59.766058 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.765982 4848 scope.go:122] "RemoveContainer" containerID="fe07862ce66ec71411925c6d6b725e76a045527023f09cb8b07aeab769e23ca8" Oct 30 21:29:59.811805 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.798360 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-home\" (UniqueName: \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-home\") pod \"24daadef-9dca-41c5-876f-9210ce533955\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " Oct 30 21:29:59.811805 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.798522 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24daadef-9dca-41c5-876f-9210ce533955-kube-api-access-t7hmx\" (UniqueName: \"kubernetes.io/projected/24daadef-9dca-41c5-876f-9210ce533955-kube-api-access-t7hmx\") pod \"24daadef-9dca-41c5-876f-9210ce533955\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " Oct 30 21:29:59.811805 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.798623 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24daadef-9dca-41c5-876f-9210ce533955-tls-certs\" (UniqueName: \"kubernetes.io/secret/24daadef-9dca-41c5-876f-9210ce533955-tls-certs\") pod \"24daadef-9dca-41c5-876f-9210ce533955\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " Oct 30 21:29:59.811805 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.798678 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-dshm\" (UniqueName: \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-dshm\") pod \"24daadef-9dca-41c5-876f-9210ce533955\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " Oct 30 21:29:59.811805 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.798761 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-tmp-dir\") pod \"24daadef-9dca-41c5-876f-9210ce533955\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " Oct 30 21:29:59.811805 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.798836 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-model-cache\") pod \"24daadef-9dca-41c5-876f-9210ce533955\" (UID: \"24daadef-9dca-41c5-876f-9210ce533955\") " Oct 30 21:29:59.834813 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.826049 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-model-cache" pod "24daadef-9dca-41c5-876f-9210ce533955" (UID: "24daadef-9dca-41c5-876f-9210ce533955"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:59.834813 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.826332 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-tmp-dir" pod "24daadef-9dca-41c5-876f-9210ce533955" (UID: "24daadef-9dca-41c5-876f-9210ce533955"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:59.834813 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.827011 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-home" pod "24daadef-9dca-41c5-876f-9210ce533955" (UID: "24daadef-9dca-41c5-876f-9210ce533955"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:59.848718 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.848648 4848 scope.go:122] "RemoveContainer" containerID="fe07862ce66ec71411925c6d6b725e76a045527023f09cb8b07aeab769e23ca8" Oct 30 21:29:59.849372 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:29:59.849249 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe07862ce66ec71411925c6d6b725e76a045527023f09cb8b07aeab769e23ca8\": container with ID starting with fe07862ce66ec71411925c6d6b725e76a045527023f09cb8b07aeab769e23ca8 not found: ID does not exist" containerID="fe07862ce66ec71411925c6d6b725e76a045527023f09cb8b07aeab769e23ca8" Oct 30 21:29:59.849372 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.849307 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe07862ce66ec71411925c6d6b725e76a045527023f09cb8b07aeab769e23ca8"} err="failed to get container status \"fe07862ce66ec71411925c6d6b725e76a045527023f09cb8b07aeab769e23ca8\": rpc error: code = NotFound desc = could not find container \"fe07862ce66ec71411925c6d6b725e76a045527023f09cb8b07aeab769e23ca8\": container with ID starting with fe07862ce66ec71411925c6d6b725e76a045527023f09cb8b07aeab769e23ca8 not found: ID does not exist" Oct 30 21:29:59.852059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.850802 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-dshm" pod "24daadef-9dca-41c5-876f-9210ce533955" (UID: "24daadef-9dca-41c5-876f-9210ce533955"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:59.852059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.850869 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24daadef-9dca-41c5-876f-9210ce533955-kube-api-access-t7hmx" pod "24daadef-9dca-41c5-876f-9210ce533955" (UID: "24daadef-9dca-41c5-876f-9210ce533955"). InnerVolumeSpecName "kube-api-access-t7hmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:59.852059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.851879 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24daadef-9dca-41c5-876f-9210ce533955-tls-certs" pod "24daadef-9dca-41c5-876f-9210ce533955" (UID: "24daadef-9dca-41c5-876f-9210ce533955"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:29:59.900763 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.900732 4848 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-tmp-dir\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:59.900763 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.900759 4848 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-model-cache\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:59.900994 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.900789 4848 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-home\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:59.900994 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.900805 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7hmx\" (UniqueName: \"kubernetes.io/projected/24daadef-9dca-41c5-876f-9210ce533955-kube-api-access-t7hmx\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:59.900994 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.900821 4848 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24daadef-9dca-41c5-876f-9210ce533955-tls-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:29:59.900994 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:29:59.900829 4848 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24daadef-9dca-41c5-876f-9210ce533955-dshm\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:00.091320 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.091284 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6"] Oct 30 21:30:00.098835 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.098760 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2ac526/test-model-2ac526-kserve-78699d86c7-ptnf6"] Oct 30 21:30:00.366564 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.366476 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm"] Oct 30 21:30:00.367590 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.367551 4848 cpu_manager.go:414] "containerMap: removing container" podUID="4cfb5025-0b7e-4697-bd4e-e65fff80e66c" containerName="istio-proxy" Oct 30 21:30:00.367590 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.367573 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cfb5025-0b7e-4697-bd4e-e65fff80e66c" containerName="istio-proxy" Oct 30 21:30:00.367853 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.367640 4848 cpu_manager.go:414] "containerMap: removing container" podUID="24daadef-9dca-41c5-876f-9210ce533955" containerName="main" Oct 30 21:30:00.367853 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.367650 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24daadef-9dca-41c5-876f-9210ce533955" containerName="main" Oct 30 21:30:00.367853 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.367686 4848 cpu_manager.go:414] "containerMap: removing container" podUID="4cfb5025-0b7e-4697-bd4e-e65fff80e66c" containerName="istio-proxy" Oct 30 21:30:00.367853 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.367694 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cfb5025-0b7e-4697-bd4e-e65fff80e66c" containerName="istio-proxy" Oct 30 21:30:00.370580 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.367878 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cfb5025-0b7e-4697-bd4e-e65fff80e66c" containerName="istio-proxy" Oct 30 21:30:00.370580 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.367898 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="8056f3a1-1b99-4dcd-8d2d-0a1a386cc1c4" containerName="istio-proxy" Oct 30 21:30:00.370580 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.367917 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="24daadef-9dca-41c5-876f-9210ce533955" containerName="main" Oct 30 21:30:00.374152 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.374129 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm" Oct 30 21:30:00.389047 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.389003 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\"" Oct 30 21:30:00.393047 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.393023 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm"] Oct 30 21:30:00.403339 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.403307 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn"] Oct 30 21:30:00.404660 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.404632 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cfb5025-0b7e-4697-bd4e-e65fff80e66c" containerName="istio-proxy" Oct 30 21:30:00.410610 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.409908 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" Oct 30 21:30:00.435546 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.435499 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn"] Oct 30 21:30:00.457412 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.457379 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69"] Oct 30 21:30:00.471969 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.471933 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69"] Oct 30 21:30:00.472138 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.472028 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69" Oct 30 21:30:00.514241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.513997 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g95nd\" (UniqueName: \"kubernetes.io/projected/28231432-c0bf-4c3b-879e-40acaceddeec-kube-api-access-g95nd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn\" (UID: \"28231432-c0bf-4c3b-879e-40acaceddeec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" Oct 30 21:30:00.514241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.514066 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c946890-dd61-47f5-8c84-ef70e09b2da4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm\" (UID: \"7c946890-dd61-47f5-8c84-ef70e09b2da4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm" Oct 30 21:30:00.514600 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.514320 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcftd\" (UniqueName: \"kubernetes.io/projected/7c946890-dd61-47f5-8c84-ef70e09b2da4-kube-api-access-lcftd\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm\" (UID: \"7c946890-dd61-47f5-8c84-ef70e09b2da4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm" Oct 30 21:30:00.514600 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.514556 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28231432-c0bf-4c3b-879e-40acaceddeec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn\" (UID: \"28231432-c0bf-4c3b-879e-40acaceddeec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" Oct 30 21:30:00.615799 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.615712 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcftd\" (UniqueName: \"kubernetes.io/projected/7c946890-dd61-47f5-8c84-ef70e09b2da4-kube-api-access-lcftd\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm\" (UID: \"7c946890-dd61-47f5-8c84-ef70e09b2da4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm" Oct 30 21:30:00.616011 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.615878 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r72n\" (UniqueName: \"kubernetes.io/projected/691eb0e8-a109-4d75-90c5-d7a7bedd9be0-kube-api-access-4r72n\") pod \"maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69\" (UID: \"691eb0e8-a109-4d75-90c5-d7a7bedd9be0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69" Oct 30 21:30:00.616011 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.615911 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28231432-c0bf-4c3b-879e-40acaceddeec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn\" (UID: \"28231432-c0bf-4c3b-879e-40acaceddeec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" Oct 30 21:30:00.616011 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.615940 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g95nd\" (UniqueName: \"kubernetes.io/projected/28231432-c0bf-4c3b-879e-40acaceddeec-kube-api-access-g95nd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn\" (UID: \"28231432-c0bf-4c3b-879e-40acaceddeec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" Oct 30 21:30:00.616011 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.615980 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c946890-dd61-47f5-8c84-ef70e09b2da4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm\" (UID: \"7c946890-dd61-47f5-8c84-ef70e09b2da4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm" Oct 30 21:30:00.616303 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.616108 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/691eb0e8-a109-4d75-90c5-d7a7bedd9be0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69\" (UID: \"691eb0e8-a109-4d75-90c5-d7a7bedd9be0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69" Oct 30 21:30:00.616912 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.616839 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c946890-dd61-47f5-8c84-ef70e09b2da4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm\" (UID: \"7c946890-dd61-47f5-8c84-ef70e09b2da4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm" Oct 30 21:30:00.616912 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.616900 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28231432-c0bf-4c3b-879e-40acaceddeec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn\" (UID: \"28231432-c0bf-4c3b-879e-40acaceddeec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" Oct 30 21:30:00.627476 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.627444 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcftd\" (UniqueName: \"kubernetes.io/projected/7c946890-dd61-47f5-8c84-ef70e09b2da4-kube-api-access-lcftd\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm\" (UID: \"7c946890-dd61-47f5-8c84-ef70e09b2da4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm" Oct 30 21:30:00.630317 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.630292 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g95nd\" (UniqueName: \"kubernetes.io/projected/28231432-c0bf-4c3b-879e-40acaceddeec-kube-api-access-g95nd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn\" (UID: \"28231432-c0bf-4c3b-879e-40acaceddeec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" Oct 30 21:30:00.708449 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.708409 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm" Oct 30 21:30:00.798864 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.737541 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r72n\" (UniqueName: \"kubernetes.io/projected/691eb0e8-a109-4d75-90c5-d7a7bedd9be0-kube-api-access-4r72n\") pod \"maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69\" (UID: \"691eb0e8-a109-4d75-90c5-d7a7bedd9be0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69" Oct 30 21:30:00.798864 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.749050 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" Oct 30 21:30:00.798864 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.750290 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/691eb0e8-a109-4d75-90c5-d7a7bedd9be0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69\" (UID: \"691eb0e8-a109-4d75-90c5-d7a7bedd9be0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69" Oct 30 21:30:00.908832 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.894707 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/691eb0e8-a109-4d75-90c5-d7a7bedd9be0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69\" (UID: \"691eb0e8-a109-4d75-90c5-d7a7bedd9be0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69" Oct 30 21:30:00.963188 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.961566 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r72n\" (UniqueName: \"kubernetes.io/projected/691eb0e8-a109-4d75-90c5-d7a7bedd9be0-kube-api-access-4r72n\") pod \"maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69\" (UID: \"691eb0e8-a109-4d75-90c5-d7a7bedd9be0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69" Oct 30 21:30:00.974827 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:00.967019 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24daadef-9dca-41c5-876f-9210ce533955" path="/var/lib/kubelet/pods/24daadef-9dca-41c5-876f-9210ce533955/volumes" Oct 30 21:30:01.097535 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:01.097275 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69" Oct 30 21:30:01.399862 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:01.382265 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm"] Oct 30 21:30:01.410088 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:30:01.410051 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28231432_c0bf_4c3b_879e_40acaceddeec.slice/crio-c0c907b17c287b7f92e1a4ff6789b399bb4b336a807f170f24ccb4d821c0fb75 WatchSource:0}: Error finding container c0c907b17c287b7f92e1a4ff6789b399bb4b336a807f170f24ccb4d821c0fb75: Status 404 returned error can't find the container with id c0c907b17c287b7f92e1a4ff6789b399bb4b336a807f170f24ccb4d821c0fb75 Oct 30 21:30:01.422022 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:01.420607 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn"] Oct 30 21:30:01.609332 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:01.609292 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69"] Oct 30 21:30:01.610714 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:30:01.610679 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691eb0e8_a109_4d75_90c5_d7a7bedd9be0.slice/crio-c376edebbafecb7fd4c7f312c16e2dd6062d094e228c02b4adb25a82ffa7cff8 WatchSource:0}: Error finding container c376edebbafecb7fd4c7f312c16e2dd6062d094e228c02b4adb25a82ffa7cff8: Status 404 returned error can't find the container with id c376edebbafecb7fd4c7f312c16e2dd6062d094e228c02b4adb25a82ffa7cff8 Oct 30 21:30:01.928143 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:01.928105 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69" event={"ID":"691eb0e8-a109-4d75-90c5-d7a7bedd9be0","Type":"ContainerStarted","Data":"c376edebbafecb7fd4c7f312c16e2dd6062d094e228c02b4adb25a82ffa7cff8"} Oct 30 21:30:01.929661 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:01.929635 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm" event={"ID":"7c946890-dd61-47f5-8c84-ef70e09b2da4","Type":"ContainerStarted","Data":"e2d7d27256d25675b4875fc49d46386adfc5c1ed2a9d0ca5a80eec322260ede5"} Oct 30 21:30:01.931514 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:01.931494 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" event={"ID":"28231432-c0bf-4c3b-879e-40acaceddeec","Type":"ContainerStarted","Data":"bc27e72ddfc2d227761b7fc909bf0688ff957d33812db692e4eb0d29ca1ffcc4"} Oct 30 21:30:01.931607 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:01.931528 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" event={"ID":"28231432-c0bf-4c3b-879e-40acaceddeec","Type":"ContainerStarted","Data":"c0c907b17c287b7f92e1a4ff6789b399bb4b336a807f170f24ccb4d821c0fb75"} Oct 30 21:30:01.950188 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:01.950130 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" podStartSLOduration=1.95011547 podStartE2EDuration="1.95011547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:01.949458495 +0000 UTC m=+3107.129355001" watchObservedRunningTime="2026-10-30 21:30:01.95011547 +0000 UTC m=+3107.130011978" Oct 30 21:30:02.946369 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:02.946333 4848 generic.go:356] "Generic (PLEG): container finished" podID="7c946890-dd61-47f5-8c84-ef70e09b2da4" containerID="2b530991c595e98365a9fd75c71ae4ece92c798248bfac51cadbefc89a470ef3" exitCode=60 Oct 30 21:30:02.946837 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:02.946409 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm" event={"ID":"7c946890-dd61-47f5-8c84-ef70e09b2da4","Type":"ContainerDied","Data":"2b530991c595e98365a9fd75c71ae4ece92c798248bfac51cadbefc89a470ef3"} Oct 30 21:30:02.946837 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:02.946675 4848 scope.go:122] "RemoveContainer" containerID="2b530991c595e98365a9fd75c71ae4ece92c798248bfac51cadbefc89a470ef3" Oct 30 21:30:02.949132 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:02.948744 4848 generic.go:356] "Generic (PLEG): container finished" podID="28231432-c0bf-4c3b-879e-40acaceddeec" containerID="bc27e72ddfc2d227761b7fc909bf0688ff957d33812db692e4eb0d29ca1ffcc4" exitCode=60 Oct 30 21:30:02.949132 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:02.949089 4848 scope.go:122] "RemoveContainer" containerID="bc27e72ddfc2d227761b7fc909bf0688ff957d33812db692e4eb0d29ca1ffcc4" Oct 30 21:30:02.949380 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:02.949162 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" event={"ID":"28231432-c0bf-4c3b-879e-40acaceddeec","Type":"ContainerDied","Data":"bc27e72ddfc2d227761b7fc909bf0688ff957d33812db692e4eb0d29ca1ffcc4"} Oct 30 21:30:02.952421 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:02.952400 4848 generic.go:356] "Generic (PLEG): container finished" podID="691eb0e8-a109-4d75-90c5-d7a7bedd9be0" containerID="6bcd76c78bb7ac6a91ec6cd6b82e983665e0cb1e0937f7e9fe9ee4771286abbc" exitCode=60 Oct 30 21:30:02.952540 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:02.952466 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69" event={"ID":"691eb0e8-a109-4d75-90c5-d7a7bedd9be0","Type":"ContainerDied","Data":"6bcd76c78bb7ac6a91ec6cd6b82e983665e0cb1e0937f7e9fe9ee4771286abbc"} Oct 30 21:30:02.952693 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:02.952678 4848 scope.go:122] "RemoveContainer" containerID="6bcd76c78bb7ac6a91ec6cd6b82e983665e0cb1e0937f7e9fe9ee4771286abbc" Oct 30 21:30:03.962047 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:03.962011 4848 generic.go:356] "Generic (PLEG): container finished" podID="691eb0e8-a109-4d75-90c5-d7a7bedd9be0" containerID="91d9930f9f5f8c7f451ab1131781852ab2adb557d55eba14738fa235e49c5de8" exitCode=60 Oct 30 21:30:03.962607 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:03.962097 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69" event={"ID":"691eb0e8-a109-4d75-90c5-d7a7bedd9be0","Type":"ContainerDied","Data":"91d9930f9f5f8c7f451ab1131781852ab2adb557d55eba14738fa235e49c5de8"} Oct 30 21:30:03.962607 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:03.962171 4848 scope.go:122] "RemoveContainer" containerID="6bcd76c78bb7ac6a91ec6cd6b82e983665e0cb1e0937f7e9fe9ee4771286abbc" Oct 30 21:30:03.962607 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:03.962499 4848 scope.go:122] "RemoveContainer" containerID="91d9930f9f5f8c7f451ab1131781852ab2adb557d55eba14738fa235e49c5de8" Oct 30 21:30:03.962974 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:30:03.962948 4848 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69_odh-ai-gateway-infra(691eb0e8-a109-4d75-90c5-d7a7bedd9be0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69" podUID="691eb0e8-a109-4d75-90c5-d7a7bedd9be0" Oct 30 21:30:03.964461 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:03.964432 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm" event={"ID":"7c946890-dd61-47f5-8c84-ef70e09b2da4","Type":"ContainerStarted","Data":"8e65a6cf0ee54f1b7b480ef9809840ae21b483c118fecb2f0657857ea13c2e6e"} Oct 30 21:30:03.966449 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:03.966423 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" event={"ID":"28231432-c0bf-4c3b-879e-40acaceddeec","Type":"ContainerStarted","Data":"991c47f87e971863bfa6f2d2c0836831a80bc6117ae39e7012ee0e6cdb61e4e3"} Oct 30 21:30:04.012624 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:04.012578 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm" podStartSLOduration=4.012564246 podStartE2EDuration="4.012564246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:04.010230786 +0000 UTC m=+3109.190127296" watchObservedRunningTime="2026-10-30 21:30:04.012564246 +0000 UTC m=+3109.192460752" Oct 30 21:30:04.975840 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:04.975804 4848 generic.go:356] "Generic (PLEG): container finished" podID="28231432-c0bf-4c3b-879e-40acaceddeec" containerID="991c47f87e971863bfa6f2d2c0836831a80bc6117ae39e7012ee0e6cdb61e4e3" exitCode=60 Oct 30 21:30:04.976345 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:04.975890 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" event={"ID":"28231432-c0bf-4c3b-879e-40acaceddeec","Type":"ContainerDied","Data":"991c47f87e971863bfa6f2d2c0836831a80bc6117ae39e7012ee0e6cdb61e4e3"} Oct 30 21:30:04.976345 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:04.975950 4848 scope.go:122] "RemoveContainer" containerID="bc27e72ddfc2d227761b7fc909bf0688ff957d33812db692e4eb0d29ca1ffcc4" Oct 30 21:30:04.976345 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:04.976268 4848 scope.go:122] "RemoveContainer" containerID="991c47f87e971863bfa6f2d2c0836831a80bc6117ae39e7012ee0e6cdb61e4e3" Oct 30 21:30:04.976715 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:30:04.976692 4848 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn_odh-ai-gateway-infra(28231432-c0bf-4c3b-879e-40acaceddeec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" podUID="28231432-c0bf-4c3b-879e-40acaceddeec" Oct 30 21:30:04.978202 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:04.978183 4848 scope.go:122] "RemoveContainer" containerID="91d9930f9f5f8c7f451ab1131781852ab2adb557d55eba14738fa235e49c5de8" Oct 30 21:30:04.978579 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:30:04.978555 4848 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69_odh-ai-gateway-infra(691eb0e8-a109-4d75-90c5-d7a7bedd9be0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69" podUID="691eb0e8-a109-4d75-90c5-d7a7bedd9be0" Oct 30 21:30:04.979929 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:04.979895 4848 generic.go:356] "Generic (PLEG): container finished" podID="7c946890-dd61-47f5-8c84-ef70e09b2da4" containerID="8e65a6cf0ee54f1b7b480ef9809840ae21b483c118fecb2f0657857ea13c2e6e" exitCode=60 Oct 30 21:30:04.980061 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:04.979991 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm" event={"ID":"7c946890-dd61-47f5-8c84-ef70e09b2da4","Type":"ContainerDied","Data":"8e65a6cf0ee54f1b7b480ef9809840ae21b483c118fecb2f0657857ea13c2e6e"} Oct 30 21:30:04.980308 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:04.980291 4848 scope.go:122] "RemoveContainer" containerID="8e65a6cf0ee54f1b7b480ef9809840ae21b483c118fecb2f0657857ea13c2e6e" Oct 30 21:30:04.980593 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:30:04.980576 4848 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm_odh-ai-gateway-infra(7c946890-dd61-47f5-8c84-ef70e09b2da4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm" podUID="7c946890-dd61-47f5-8c84-ef70e09b2da4" Oct 30 21:30:04.989596 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:04.989570 4848 scope.go:122] "RemoveContainer" containerID="2b530991c595e98365a9fd75c71ae4ece92c798248bfac51cadbefc89a470ef3" Oct 30 21:30:06.076790 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:06.070678 4848 scope.go:122] "RemoveContainer" containerID="8e65a6cf0ee54f1b7b480ef9809840ae21b483c118fecb2f0657857ea13c2e6e" Oct 30 21:30:06.076790 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:30:06.071107 4848 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm_odh-ai-gateway-infra(7c946890-dd61-47f5-8c84-ef70e09b2da4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm" podUID="7c946890-dd61-47f5-8c84-ef70e09b2da4" Oct 30 21:30:06.083811 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:06.083621 4848 scope.go:122] "RemoveContainer" containerID="991c47f87e971863bfa6f2d2c0836831a80bc6117ae39e7012ee0e6cdb61e4e3" Oct 30 21:30:06.084956 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:30:06.084129 4848 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn_odh-ai-gateway-infra(28231432-c0bf-4c3b-879e-40acaceddeec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" podUID="28231432-c0bf-4c3b-879e-40acaceddeec" Oct 30 21:30:10.359874 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:10.359829 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" podUID="57e19834-f4ba-495f-a5a8-2b73f8e36cb8" containerName="main" containerID="cri-o://8d4fec84f2979e4dc9e66d392b8afac418abfdbee69735c3e96275f75f91f4e1" gracePeriod=45 Oct 30 21:30:10.648141 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:10.648115 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:30:10.814491 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:10.814464 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-kube-api-access-dcws2\" (UniqueName: \"kubernetes.io/projected/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-kube-api-access-dcws2\") pod \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " Oct 30 21:30:10.814666 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:10.814530 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-model-cache\") pod \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " Oct 30 21:30:10.814666 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:10.814612 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-tmp-dir\") pod \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " Oct 30 21:30:10.814839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:10.814721 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-tls-certs\" (UniqueName: \"kubernetes.io/secret/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-tls-certs\") pod \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " Oct 30 21:30:10.814839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:10.814799 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-home\" (UniqueName: \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-home\") pod \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " Oct 30 21:30:10.814967 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:10.814879 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-dshm\") pod \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\" (UID: \"57e19834-f4ba-495f-a5a8-2b73f8e36cb8\") " Oct 30 21:30:10.815183 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:10.814914 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-model-cache" pod "57e19834-f4ba-495f-a5a8-2b73f8e36cb8" (UID: "57e19834-f4ba-495f-a5a8-2b73f8e36cb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:10.815334 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:10.814926 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-tmp-dir" pod "57e19834-f4ba-495f-a5a8-2b73f8e36cb8" (UID: "57e19834-f4ba-495f-a5a8-2b73f8e36cb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:10.815334 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:10.815045 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-home" pod "57e19834-f4ba-495f-a5a8-2b73f8e36cb8" (UID: "57e19834-f4ba-495f-a5a8-2b73f8e36cb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:10.815949 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:10.815919 4848 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-home\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:10.816064 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:10.815953 4848 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-model-cache\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:10.816064 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:10.815970 4848 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-tmp-dir\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:10.817113 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:10.817086 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-tls-certs" pod "57e19834-f4ba-495f-a5a8-2b73f8e36cb8" (UID: "57e19834-f4ba-495f-a5a8-2b73f8e36cb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:10.817233 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:10.817180 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-dshm" pod "57e19834-f4ba-495f-a5a8-2b73f8e36cb8" (UID: "57e19834-f4ba-495f-a5a8-2b73f8e36cb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:10.817233 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:10.817225 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-kube-api-access-dcws2" pod "57e19834-f4ba-495f-a5a8-2b73f8e36cb8" (UID: "57e19834-f4ba-495f-a5a8-2b73f8e36cb8"). InnerVolumeSpecName "kube-api-access-dcws2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:10.917185 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:10.917158 4848 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-dshm\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:10.917185 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:10.917183 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcws2\" (UniqueName: \"kubernetes.io/projected/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-kube-api-access-dcws2\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:10.917356 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:10.917194 4848 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57e19834-f4ba-495f-a5a8-2b73f8e36cb8-tls-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:11.122668 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:11.122573 4848 generic.go:356] "Generic (PLEG): container finished" podID="57e19834-f4ba-495f-a5a8-2b73f8e36cb8" containerID="8d4fec84f2979e4dc9e66d392b8afac418abfdbee69735c3e96275f75f91f4e1" exitCode=0 Oct 30 21:30:11.122668 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:11.122620 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" Oct 30 21:30:11.122911 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:11.122655 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" event={"ID":"57e19834-f4ba-495f-a5a8-2b73f8e36cb8","Type":"ContainerDied","Data":"8d4fec84f2979e4dc9e66d392b8afac418abfdbee69735c3e96275f75f91f4e1"} Oct 30 21:30:11.122911 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:11.122722 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x" event={"ID":"57e19834-f4ba-495f-a5a8-2b73f8e36cb8","Type":"ContainerDied","Data":"1f47a6dd20994aba35e523876226c201028012fb72dc2a07b2139a2a35dbe77f"} Oct 30 21:30:11.122911 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:11.122747 4848 scope.go:122] "RemoveContainer" containerID="8d4fec84f2979e4dc9e66d392b8afac418abfdbee69735c3e96275f75f91f4e1" Oct 30 21:30:11.135642 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:11.135625 4848 scope.go:122] "RemoveContainer" containerID="8d4fec84f2979e4dc9e66d392b8afac418abfdbee69735c3e96275f75f91f4e1" Oct 30 21:30:11.135976 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:30:11.135952 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4fec84f2979e4dc9e66d392b8afac418abfdbee69735c3e96275f75f91f4e1\": container with ID starting with 8d4fec84f2979e4dc9e66d392b8afac418abfdbee69735c3e96275f75f91f4e1 not found: ID does not exist" containerID="8d4fec84f2979e4dc9e66d392b8afac418abfdbee69735c3e96275f75f91f4e1" Oct 30 21:30:11.136074 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:11.135984 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4fec84f2979e4dc9e66d392b8afac418abfdbee69735c3e96275f75f91f4e1"} err="failed to get container status \"8d4fec84f2979e4dc9e66d392b8afac418abfdbee69735c3e96275f75f91f4e1\": rpc error: code = NotFound desc = could not find container \"8d4fec84f2979e4dc9e66d392b8afac418abfdbee69735c3e96275f75f91f4e1\": container with ID starting with 8d4fec84f2979e4dc9e66d392b8afac418abfdbee69735c3e96275f75f91f4e1 not found: ID does not exist" Oct 30 21:30:11.145241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:11.145214 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x"] Oct 30 21:30:11.153068 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:11.153047 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a3fb70/test-model-a3fb70-kserve-7789b69865-4m55x"] Oct 30 21:30:12.936400 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:12.936316 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57e19834-f4ba-495f-a5a8-2b73f8e36cb8" path="/var/lib/kubelet/pods/57e19834-f4ba-495f-a5a8-2b73f8e36cb8/volumes" Oct 30 21:30:14.444138 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:14.444099 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq"] Oct 30 21:30:14.444670 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:14.444361 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" podUID="01a47632-b985-464f-8ee1-4fe60a1b3258" containerName="istio-proxy" containerID="cri-o://9961db2fc9b44abf0a714d9150b821813502338cedeeefa564ef085cdb3ad483" gracePeriod=30 Oct 30 21:30:17.925589 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:17.925547 4848 scope.go:122] "RemoveContainer" containerID="8e65a6cf0ee54f1b7b480ef9809840ae21b483c118fecb2f0657857ea13c2e6e" Oct 30 21:30:17.926091 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:17.926072 4848 scope.go:122] "RemoveContainer" containerID="991c47f87e971863bfa6f2d2c0836831a80bc6117ae39e7012ee0e6cdb61e4e3" Oct 30 21:30:17.926325 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:17.926295 4848 scope.go:122] "RemoveContainer" containerID="91d9930f9f5f8c7f451ab1131781852ab2adb557d55eba14738fa235e49c5de8" Oct 30 21:30:19.200505 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.200467 4848 generic.go:356] "Generic (PLEG): container finished" podID="28231432-c0bf-4c3b-879e-40acaceddeec" containerID="1b7f348539b8d15c6672a6c3c4c367ee7c24165b314b0a184008fadb2fa27453" exitCode=60 Oct 30 21:30:19.201033 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.200640 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" event={"ID":"28231432-c0bf-4c3b-879e-40acaceddeec","Type":"ContainerDied","Data":"1b7f348539b8d15c6672a6c3c4c367ee7c24165b314b0a184008fadb2fa27453"} Oct 30 21:30:19.201033 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.200697 4848 scope.go:122] "RemoveContainer" containerID="991c47f87e971863bfa6f2d2c0836831a80bc6117ae39e7012ee0e6cdb61e4e3" Oct 30 21:30:19.201480 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.201209 4848 scope.go:122] "RemoveContainer" containerID="1b7f348539b8d15c6672a6c3c4c367ee7c24165b314b0a184008fadb2fa27453" Oct 30 21:30:19.201576 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:30:19.201550 4848 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn_odh-ai-gateway-infra(28231432-c0bf-4c3b-879e-40acaceddeec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" podUID="28231432-c0bf-4c3b-879e-40acaceddeec" Oct 30 21:30:19.204385 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.204355 4848 generic.go:356] "Generic (PLEG): container finished" podID="691eb0e8-a109-4d75-90c5-d7a7bedd9be0" containerID="81bc2432b01f938bd5625c980a39dfead916cabdc1d08fdfe7d72ae678d6003d" exitCode=60 Oct 30 21:30:19.204497 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.204389 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69" event={"ID":"691eb0e8-a109-4d75-90c5-d7a7bedd9be0","Type":"ContainerDied","Data":"81bc2432b01f938bd5625c980a39dfead916cabdc1d08fdfe7d72ae678d6003d"} Oct 30 21:30:19.205757 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.205730 4848 scope.go:122] "RemoveContainer" containerID="81bc2432b01f938bd5625c980a39dfead916cabdc1d08fdfe7d72ae678d6003d" Oct 30 21:30:19.206199 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:30:19.206166 4848 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69_odh-ai-gateway-infra(691eb0e8-a109-4d75-90c5-d7a7bedd9be0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69" podUID="691eb0e8-a109-4d75-90c5-d7a7bedd9be0" Oct 30 21:30:19.207029 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.207005 4848 generic.go:356] "Generic (PLEG): container finished" podID="7c946890-dd61-47f5-8c84-ef70e09b2da4" containerID="ac575e47b0f0e0ece8910df5767db0bdc50236453af10c5db45f19b0edcaf87e" exitCode=60 Oct 30 21:30:19.207154 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.207069 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm" event={"ID":"7c946890-dd61-47f5-8c84-ef70e09b2da4","Type":"ContainerDied","Data":"ac575e47b0f0e0ece8910df5767db0bdc50236453af10c5db45f19b0edcaf87e"} Oct 30 21:30:19.207488 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.207470 4848 scope.go:122] "RemoveContainer" containerID="ac575e47b0f0e0ece8910df5767db0bdc50236453af10c5db45f19b0edcaf87e" Oct 30 21:30:19.207907 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:30:19.207854 4848 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm_odh-ai-gateway-infra(7c946890-dd61-47f5-8c84-ef70e09b2da4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm" podUID="7c946890-dd61-47f5-8c84-ef70e09b2da4" Oct 30 21:30:19.227172 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.223417 4848 scope.go:122] "RemoveContainer" containerID="91d9930f9f5f8c7f451ab1131781852ab2adb557d55eba14738fa235e49c5de8" Oct 30 21:30:19.241272 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.241245 4848 scope.go:122] "RemoveContainer" containerID="8e65a6cf0ee54f1b7b480ef9809840ae21b483c118fecb2f0657857ea13c2e6e" Oct 30 21:30:19.739342 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.739318 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:30:19.921794 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.921730 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-token\" (UniqueName: \"kubernetes.io/projected/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-token\") pod \"01a47632-b985-464f-8ee1-4fe60a1b3258\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " Oct 30 21:30:19.921985 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.921823 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-workload-socket\") pod \"01a47632-b985-464f-8ee1-4fe60a1b3258\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " Oct 30 21:30:19.921985 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.921877 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-podinfo\") pod \"01a47632-b985-464f-8ee1-4fe60a1b3258\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " Oct 30 21:30:19.921985 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.921914 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-envoy\") pod \"01a47632-b985-464f-8ee1-4fe60a1b3258\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " Oct 30 21:30:19.921985 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.921961 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a47632-b985-464f-8ee1-4fe60a1b3258-kube-api-access-ngpmh\" (UniqueName: \"kubernetes.io/projected/01a47632-b985-464f-8ee1-4fe60a1b3258-kube-api-access-ngpmh\") pod \"01a47632-b985-464f-8ee1-4fe60a1b3258\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " Oct 30 21:30:19.922235 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.922042 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-workload-certs\") pod \"01a47632-b985-464f-8ee1-4fe60a1b3258\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " Oct 30 21:30:19.922235 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.922176 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-credential-socket\") pod \"01a47632-b985-464f-8ee1-4fe60a1b3258\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " Oct 30 21:30:19.998892 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.922372 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-data\") pod \"01a47632-b985-464f-8ee1-4fe60a1b3258\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " Oct 30 21:30:19.998892 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.922425 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a47632-b985-464f-8ee1-4fe60a1b3258-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a47632-b985-464f-8ee1-4fe60a1b3258-istiod-ca-cert\") pod \"01a47632-b985-464f-8ee1-4fe60a1b3258\" (UID: \"01a47632-b985-464f-8ee1-4fe60a1b3258\") " Oct 30 21:30:19.998892 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.970101 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-token" pod "01a47632-b985-464f-8ee1-4fe60a1b3258" (UID: "01a47632-b985-464f-8ee1-4fe60a1b3258"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:19.998892 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.971099 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-workload-certs" pod "01a47632-b985-464f-8ee1-4fe60a1b3258" (UID: "01a47632-b985-464f-8ee1-4fe60a1b3258"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:19.998892 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.971452 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-data" pod "01a47632-b985-464f-8ee1-4fe60a1b3258" (UID: "01a47632-b985-464f-8ee1-4fe60a1b3258"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:19.998892 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.971685 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-credential-socket" pod "01a47632-b985-464f-8ee1-4fe60a1b3258" (UID: "01a47632-b985-464f-8ee1-4fe60a1b3258"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:19.998892 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.972100 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a47632-b985-464f-8ee1-4fe60a1b3258-istiod-ca-cert" pod "01a47632-b985-464f-8ee1-4fe60a1b3258" (UID: "01a47632-b985-464f-8ee1-4fe60a1b3258"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:19.998892 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:19.973278 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-workload-socket" pod "01a47632-b985-464f-8ee1-4fe60a1b3258" (UID: "01a47632-b985-464f-8ee1-4fe60a1b3258"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:20.024724 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.024691 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:20.024858 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.024731 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01a47632-b985-464f-8ee1-4fe60a1b3258-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:20.024858 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.024751 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:20.024858 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.024779 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:20.024858 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.024792 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:20.024858 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.024806 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:20.035354 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.035319 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-podinfo" pod "01a47632-b985-464f-8ee1-4fe60a1b3258" (UID: "01a47632-b985-464f-8ee1-4fe60a1b3258"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:30:20.035517 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.035345 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-envoy" pod "01a47632-b985-464f-8ee1-4fe60a1b3258" (UID: "01a47632-b985-464f-8ee1-4fe60a1b3258"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:20.045073 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.045023 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a47632-b985-464f-8ee1-4fe60a1b3258-kube-api-access-ngpmh" pod "01a47632-b985-464f-8ee1-4fe60a1b3258" (UID: "01a47632-b985-464f-8ee1-4fe60a1b3258"). InnerVolumeSpecName "kube-api-access-ngpmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:20.126212 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.126176 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:20.126212 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.126210 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01a47632-b985-464f-8ee1-4fe60a1b3258-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:20.126402 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.126223 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngpmh\" (UniqueName: \"kubernetes.io/projected/01a47632-b985-464f-8ee1-4fe60a1b3258-kube-api-access-ngpmh\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:20.218257 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.218225 4848 generic.go:356] "Generic (PLEG): container finished" podID="01a47632-b985-464f-8ee1-4fe60a1b3258" containerID="9961db2fc9b44abf0a714d9150b821813502338cedeeefa564ef085cdb3ad483" exitCode=0 Oct 30 21:30:20.218748 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.218282 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" Oct 30 21:30:20.218748 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.218301 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" event={"ID":"01a47632-b985-464f-8ee1-4fe60a1b3258","Type":"ContainerDied","Data":"9961db2fc9b44abf0a714d9150b821813502338cedeeefa564ef085cdb3ad483"} Oct 30 21:30:20.218748 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.218367 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq" event={"ID":"01a47632-b985-464f-8ee1-4fe60a1b3258","Type":"ContainerDied","Data":"ef4ae02c98e099252291635ed6eeed19452779c14d4a53330d8a61f18562398b"} Oct 30 21:30:20.218748 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.218392 4848 scope.go:122] "RemoveContainer" containerID="9961db2fc9b44abf0a714d9150b821813502338cedeeefa564ef085cdb3ad483" Oct 30 21:30:20.232911 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.232893 4848 scope.go:122] "RemoveContainer" containerID="6adfe6528c1330c09fabb1d8b4ab1f784850138591c7147430f5c3f18fba47f5" Oct 30 21:30:20.234489 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.234467 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn"] Oct 30 21:30:20.242725 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.242707 4848 scope.go:122] "RemoveContainer" containerID="9961db2fc9b44abf0a714d9150b821813502338cedeeefa564ef085cdb3ad483" Oct 30 21:30:20.243062 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:30:20.243041 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9961db2fc9b44abf0a714d9150b821813502338cedeeefa564ef085cdb3ad483\": container with ID starting with 9961db2fc9b44abf0a714d9150b821813502338cedeeefa564ef085cdb3ad483 not found: ID does not exist" containerID="9961db2fc9b44abf0a714d9150b821813502338cedeeefa564ef085cdb3ad483" Oct 30 21:30:20.243121 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.243071 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9961db2fc9b44abf0a714d9150b821813502338cedeeefa564ef085cdb3ad483"} err="failed to get container status \"9961db2fc9b44abf0a714d9150b821813502338cedeeefa564ef085cdb3ad483\": rpc error: code = NotFound desc = could not find container \"9961db2fc9b44abf0a714d9150b821813502338cedeeefa564ef085cdb3ad483\": container with ID starting with 9961db2fc9b44abf0a714d9150b821813502338cedeeefa564ef085cdb3ad483 not found: ID does not exist" Oct 30 21:30:20.243121 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.243095 4848 scope.go:122] "RemoveContainer" containerID="6adfe6528c1330c09fabb1d8b4ab1f784850138591c7147430f5c3f18fba47f5" Oct 30 21:30:20.243369 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:30:20.243349 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6adfe6528c1330c09fabb1d8b4ab1f784850138591c7147430f5c3f18fba47f5\": container with ID starting with 6adfe6528c1330c09fabb1d8b4ab1f784850138591c7147430f5c3f18fba47f5 not found: ID does not exist" containerID="6adfe6528c1330c09fabb1d8b4ab1f784850138591c7147430f5c3f18fba47f5" Oct 30 21:30:20.243415 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.243377 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6adfe6528c1330c09fabb1d8b4ab1f784850138591c7147430f5c3f18fba47f5"} err="failed to get container status \"6adfe6528c1330c09fabb1d8b4ab1f784850138591c7147430f5c3f18fba47f5\": rpc error: code = NotFound desc = could not find container \"6adfe6528c1330c09fabb1d8b4ab1f784850138591c7147430f5c3f18fba47f5\": container with ID starting with 6adfe6528c1330c09fabb1d8b4ab1f784850138591c7147430f5c3f18fba47f5 not found: ID does not exist" Oct 30 21:30:20.277186 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.277099 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69"] Oct 30 21:30:20.293177 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.287457 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm"] Oct 30 21:30:20.308825 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.300038 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq"] Oct 30 21:30:20.308825 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.307161 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-321ccd-openshift-default-787dff9cd7-mxtdq"] Oct 30 21:30:20.451378 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.451347 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" Oct 30 21:30:20.512695 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.512669 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69" Oct 30 21:30:20.534352 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.534275 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm" Oct 30 21:30:20.534588 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.534569 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28231432-c0bf-4c3b-879e-40acaceddeec-kube-api-access-g95nd\" (UniqueName: \"kubernetes.io/projected/28231432-c0bf-4c3b-879e-40acaceddeec-kube-api-access-g95nd\") pod \"28231432-c0bf-4c3b-879e-40acaceddeec\" (UID: \"28231432-c0bf-4c3b-879e-40acaceddeec\") " Oct 30 21:30:20.534674 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.534656 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/691eb0e8-a109-4d75-90c5-d7a7bedd9be0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/691eb0e8-a109-4d75-90c5-d7a7bedd9be0-maas-api-service-ca\") pod \"691eb0e8-a109-4d75-90c5-d7a7bedd9be0\" (UID: \"691eb0e8-a109-4d75-90c5-d7a7bedd9be0\") " Oct 30 21:30:20.534737 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.534707 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28231432-c0bf-4c3b-879e-40acaceddeec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28231432-c0bf-4c3b-879e-40acaceddeec-maas-api-service-ca\") pod \"28231432-c0bf-4c3b-879e-40acaceddeec\" (UID: \"28231432-c0bf-4c3b-879e-40acaceddeec\") " Oct 30 21:30:20.534884 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.534866 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/691eb0e8-a109-4d75-90c5-d7a7bedd9be0-kube-api-access-4r72n\" (UniqueName: \"kubernetes.io/projected/691eb0e8-a109-4d75-90c5-d7a7bedd9be0-kube-api-access-4r72n\") pod \"691eb0e8-a109-4d75-90c5-d7a7bedd9be0\" (UID: \"691eb0e8-a109-4d75-90c5-d7a7bedd9be0\") " Oct 30 21:30:20.535149 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.535113 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/691eb0e8-a109-4d75-90c5-d7a7bedd9be0-maas-api-service-ca" pod "691eb0e8-a109-4d75-90c5-d7a7bedd9be0" (UID: "691eb0e8-a109-4d75-90c5-d7a7bedd9be0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:20.538427 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.535433 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28231432-c0bf-4c3b-879e-40acaceddeec-maas-api-service-ca" pod "28231432-c0bf-4c3b-879e-40acaceddeec" (UID: "28231432-c0bf-4c3b-879e-40acaceddeec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:20.538427 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.535481 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/691eb0e8-a109-4d75-90c5-d7a7bedd9be0-maas-api-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:20.538427 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.537678 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28231432-c0bf-4c3b-879e-40acaceddeec-kube-api-access-g95nd" pod "28231432-c0bf-4c3b-879e-40acaceddeec" (UID: "28231432-c0bf-4c3b-879e-40acaceddeec"). InnerVolumeSpecName "kube-api-access-g95nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:20.538427 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.537810 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691eb0e8-a109-4d75-90c5-d7a7bedd9be0-kube-api-access-4r72n" pod "691eb0e8-a109-4d75-90c5-d7a7bedd9be0" (UID: "691eb0e8-a109-4d75-90c5-d7a7bedd9be0"). InnerVolumeSpecName "kube-api-access-4r72n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:20.636939 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.636895 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c946890-dd61-47f5-8c84-ef70e09b2da4-kube-api-access-lcftd\" (UniqueName: \"kubernetes.io/projected/7c946890-dd61-47f5-8c84-ef70e09b2da4-kube-api-access-lcftd\") pod \"7c946890-dd61-47f5-8c84-ef70e09b2da4\" (UID: \"7c946890-dd61-47f5-8c84-ef70e09b2da4\") " Oct 30 21:30:20.637132 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.636966 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c946890-dd61-47f5-8c84-ef70e09b2da4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c946890-dd61-47f5-8c84-ef70e09b2da4-maas-api-service-ca\") pod \"7c946890-dd61-47f5-8c84-ef70e09b2da4\" (UID: \"7c946890-dd61-47f5-8c84-ef70e09b2da4\") " Oct 30 21:30:20.637357 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.637332 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c946890-dd61-47f5-8c84-ef70e09b2da4-maas-api-service-ca" pod "7c946890-dd61-47f5-8c84-ef70e09b2da4" (UID: "7c946890-dd61-47f5-8c84-ef70e09b2da4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:20.637585 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.637569 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g95nd\" (UniqueName: \"kubernetes.io/projected/28231432-c0bf-4c3b-879e-40acaceddeec-kube-api-access-g95nd\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:20.637655 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.637591 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c946890-dd61-47f5-8c84-ef70e09b2da4-maas-api-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:20.637655 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.637605 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28231432-c0bf-4c3b-879e-40acaceddeec-maas-api-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:20.637655 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.637619 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r72n\" (UniqueName: \"kubernetes.io/projected/691eb0e8-a109-4d75-90c5-d7a7bedd9be0-kube-api-access-4r72n\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:20.638962 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.638943 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c946890-dd61-47f5-8c84-ef70e09b2da4-kube-api-access-lcftd" pod "7c946890-dd61-47f5-8c84-ef70e09b2da4" (UID: "7c946890-dd61-47f5-8c84-ef70e09b2da4"). InnerVolumeSpecName "kube-api-access-lcftd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:20.782300 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.782260 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcftd\" (UniqueName: \"kubernetes.io/projected/7c946890-dd61-47f5-8c84-ef70e09b2da4-kube-api-access-lcftd\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:20.934251 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:20.934219 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a47632-b985-464f-8ee1-4fe60a1b3258" path="/var/lib/kubelet/pods/01a47632-b985-464f-8ee1-4fe60a1b3258/volumes" Oct 30 21:30:21.240489 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:21.240401 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69" Oct 30 21:30:21.240489 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:21.240440 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69" event={"ID":"691eb0e8-a109-4d75-90c5-d7a7bedd9be0","Type":"ContainerDied","Data":"c376edebbafecb7fd4c7f312c16e2dd6062d094e228c02b4adb25a82ffa7cff8"} Oct 30 21:30:21.241054 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:21.240510 4848 scope.go:122] "RemoveContainer" containerID="81bc2432b01f938bd5625c980a39dfead916cabdc1d08fdfe7d72ae678d6003d" Oct 30 21:30:21.242072 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:21.242049 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm" Oct 30 21:30:21.242191 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:21.242055 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm" event={"ID":"7c946890-dd61-47f5-8c84-ef70e09b2da4","Type":"ContainerDied","Data":"e2d7d27256d25675b4875fc49d46386adfc5c1ed2a9d0ca5a80eec322260ede5"} Oct 30 21:30:21.245228 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:21.245204 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" Oct 30 21:30:21.245316 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:21.245202 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn" event={"ID":"28231432-c0bf-4c3b-879e-40acaceddeec","Type":"ContainerDied","Data":"c0c907b17c287b7f92e1a4ff6789b399bb4b336a807f170f24ccb4d821c0fb75"} Oct 30 21:30:21.252214 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:21.252192 4848 scope.go:122] "RemoveContainer" containerID="ac575e47b0f0e0ece8910df5767db0bdc50236453af10c5db45f19b0edcaf87e" Oct 30 21:30:21.265422 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:21.265397 4848 scope.go:122] "RemoveContainer" containerID="1b7f348539b8d15c6672a6c3c4c367ee7c24165b314b0a184008fadb2fa27453" Oct 30 21:30:21.298025 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:21.297987 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn"] Oct 30 21:30:21.301190 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:21.301165 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3a65ec-29889930-qwvsn"] Oct 30 21:30:21.311311 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:21.311284 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69"] Oct 30 21:30:21.319603 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:21.319581 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-00909f-29889930-gzq69"] Oct 30 21:30:21.339948 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:21.339922 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm"] Oct 30 21:30:21.346597 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:21.346573 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-a50675-29889930-zbxtm"] Oct 30 21:30:22.511378 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:30:22.511341 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:30:22.934762 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:22.934733 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28231432-c0bf-4c3b-879e-40acaceddeec" path="/var/lib/kubelet/pods/28231432-c0bf-4c3b-879e-40acaceddeec/volumes" Oct 30 21:30:22.935235 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:22.935220 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="691eb0e8-a109-4d75-90c5-d7a7bedd9be0" path="/var/lib/kubelet/pods/691eb0e8-a109-4d75-90c5-d7a7bedd9be0/volumes" Oct 30 21:30:22.935614 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:22.935601 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c946890-dd61-47f5-8c84-ef70e09b2da4" path="/var/lib/kubelet/pods/7c946890-dd61-47f5-8c84-ef70e09b2da4/volumes" Oct 30 21:30:39.447115 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:39.447078 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck"] Oct 30 21:30:39.447947 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:39.447375 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" podUID="906f25cf-0aea-484b-9dcc-5302dfb1c5e4" containerName="maas-api" containerID="cri-o://6d7ffcdd1bd03b3d28383a05398c9cb16d492000b625595c4374ced043b1a899" gracePeriod=30 Oct 30 21:30:39.732371 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:39.732340 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" Oct 30 21:30:39.852948 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:39.852915 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-maas-api-tls\") pod \"906f25cf-0aea-484b-9dcc-5302dfb1c5e4\" (UID: \"906f25cf-0aea-484b-9dcc-5302dfb1c5e4\") " Oct 30 21:30:39.853123 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:39.852979 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-kube-api-access-v9f4z\" (UniqueName: \"kubernetes.io/projected/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-kube-api-access-v9f4z\") pod \"906f25cf-0aea-484b-9dcc-5302dfb1c5e4\" (UID: \"906f25cf-0aea-484b-9dcc-5302dfb1c5e4\") " Oct 30 21:30:39.853123 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:39.853015 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-openshift-service-ca\") pod \"906f25cf-0aea-484b-9dcc-5302dfb1c5e4\" (UID: \"906f25cf-0aea-484b-9dcc-5302dfb1c5e4\") " Oct 30 21:30:39.853294 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:39.853277 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-metrics-certs\") pod \"906f25cf-0aea-484b-9dcc-5302dfb1c5e4\" (UID: \"906f25cf-0aea-484b-9dcc-5302dfb1c5e4\") " Oct 30 21:30:39.853473 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:39.853446 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-openshift-service-ca" pod "906f25cf-0aea-484b-9dcc-5302dfb1c5e4" (UID: "906f25cf-0aea-484b-9dcc-5302dfb1c5e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:39.854028 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:39.853999 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:39.855452 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:39.855422 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-kube-api-access-v9f4z" pod "906f25cf-0aea-484b-9dcc-5302dfb1c5e4" (UID: "906f25cf-0aea-484b-9dcc-5302dfb1c5e4"). InnerVolumeSpecName "kube-api-access-v9f4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:39.855603 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:39.855559 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-maas-api-tls" pod "906f25cf-0aea-484b-9dcc-5302dfb1c5e4" (UID: "906f25cf-0aea-484b-9dcc-5302dfb1c5e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:39.855671 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:39.855655 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-metrics-certs" pod "906f25cf-0aea-484b-9dcc-5302dfb1c5e4" (UID: "906f25cf-0aea-484b-9dcc-5302dfb1c5e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:30:39.955954 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:39.955879 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9f4z\" (UniqueName: \"kubernetes.io/projected/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-kube-api-access-v9f4z\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:39.955954 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:39.955909 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:39.955954 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:39.955921 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/906f25cf-0aea-484b-9dcc-5302dfb1c5e4-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:40.432167 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:40.426152 4848 generic.go:356] "Generic (PLEG): container finished" podID="906f25cf-0aea-484b-9dcc-5302dfb1c5e4" containerID="6d7ffcdd1bd03b3d28383a05398c9cb16d492000b625595c4374ced043b1a899" exitCode=0 Oct 30 21:30:40.432167 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:40.426418 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" event={"ID":"906f25cf-0aea-484b-9dcc-5302dfb1c5e4","Type":"ContainerDied","Data":"6d7ffcdd1bd03b3d28383a05398c9cb16d492000b625595c4374ced043b1a899"} Oct 30 21:30:40.432167 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:40.426485 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" event={"ID":"906f25cf-0aea-484b-9dcc-5302dfb1c5e4","Type":"ContainerDied","Data":"3016be2d2959606034866009f6b1f2edc2596da6ebcd88faf4cb81c8619cca42"} Oct 30 21:30:40.432167 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:40.426520 4848 scope.go:122] "RemoveContainer" containerID="6d7ffcdd1bd03b3d28383a05398c9cb16d492000b625595c4374ced043b1a899" Oct 30 21:30:40.432167 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:40.426699 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck" Oct 30 21:30:40.462974 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:40.462939 4848 scope.go:122] "RemoveContainer" containerID="6d7ffcdd1bd03b3d28383a05398c9cb16d492000b625595c4374ced043b1a899" Oct 30 21:30:40.463478 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:30:40.463437 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d7ffcdd1bd03b3d28383a05398c9cb16d492000b625595c4374ced043b1a899\": container with ID starting with 6d7ffcdd1bd03b3d28383a05398c9cb16d492000b625595c4374ced043b1a899 not found: ID does not exist" containerID="6d7ffcdd1bd03b3d28383a05398c9cb16d492000b625595c4374ced043b1a899" Oct 30 21:30:40.463606 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:40.463489 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d7ffcdd1bd03b3d28383a05398c9cb16d492000b625595c4374ced043b1a899"} err="failed to get container status \"6d7ffcdd1bd03b3d28383a05398c9cb16d492000b625595c4374ced043b1a899\": rpc error: code = NotFound desc = could not find container \"6d7ffcdd1bd03b3d28383a05398c9cb16d492000b625595c4374ced043b1a899\": container with ID starting with 6d7ffcdd1bd03b3d28383a05398c9cb16d492000b625595c4374ced043b1a899 not found: ID does not exist" Oct 30 21:30:40.475318 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:40.475296 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck"] Oct 30 21:30:40.479835 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:40.479815 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9b78e4-59cf869b88-48bck"] Oct 30 21:30:40.934928 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:40.934894 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="906f25cf-0aea-484b-9dcc-5302dfb1c5e4" path="/var/lib/kubelet/pods/906f25cf-0aea-484b-9dcc-5302dfb1c5e4/volumes" Oct 30 21:30:48.536536 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:48.536495 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf"] Oct 30 21:30:48.537073 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:48.536820 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" podUID="b02fca84-f078-4d46-a842-a9b29f63eaaa" containerName="istio-proxy" containerID="cri-o://4ce63e6a98d91d742876bc882581170a7fc53b000a45b9fdc6e028d47cd7d7c9" gracePeriod=30 Oct 30 21:30:52.145404 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:52.145364 4848 patch_prober.go:29] interesting pod/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:30:52.145867 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:52.145440 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" podUID="b02fca84-f078-4d46-a842-a9b29f63eaaa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:30:52.511309 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:30:52.511267 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:30:53.827170 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.827143 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:30:53.933615 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.933587 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-workload-certs\") pod \"b02fca84-f078-4d46-a842-a9b29f63eaaa\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " Oct 30 21:30:53.933834 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.933629 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-token\" (UniqueName: \"kubernetes.io/projected/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-token\") pod \"b02fca84-f078-4d46-a842-a9b29f63eaaa\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " Oct 30 21:30:53.933834 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.933686 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-credential-socket\") pod \"b02fca84-f078-4d46-a842-a9b29f63eaaa\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " Oct 30 21:30:53.933834 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.933812 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-podinfo\") pod \"b02fca84-f078-4d46-a842-a9b29f63eaaa\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " Oct 30 21:30:53.934032 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.933849 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02fca84-f078-4d46-a842-a9b29f63eaaa-kube-api-access-bjdt6\" (UniqueName: \"kubernetes.io/projected/b02fca84-f078-4d46-a842-a9b29f63eaaa-kube-api-access-bjdt6\") pod \"b02fca84-f078-4d46-a842-a9b29f63eaaa\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " Oct 30 21:30:53.934032 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.933893 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-envoy\") pod \"b02fca84-f078-4d46-a842-a9b29f63eaaa\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " Oct 30 21:30:53.934032 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.933939 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-workload-certs" pod "b02fca84-f078-4d46-a842-a9b29f63eaaa" (UID: "b02fca84-f078-4d46-a842-a9b29f63eaaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:53.934032 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.933976 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-data\") pod \"b02fca84-f078-4d46-a842-a9b29f63eaaa\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " Oct 30 21:30:53.934256 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.934034 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-credential-socket" pod "b02fca84-f078-4d46-a842-a9b29f63eaaa" (UID: "b02fca84-f078-4d46-a842-a9b29f63eaaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:53.934256 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.934123 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-workload-socket\") pod \"b02fca84-f078-4d46-a842-a9b29f63eaaa\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " Oct 30 21:30:53.934256 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.934157 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b02fca84-f078-4d46-a842-a9b29f63eaaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b02fca84-f078-4d46-a842-a9b29f63eaaa-istiod-ca-cert\") pod \"b02fca84-f078-4d46-a842-a9b29f63eaaa\" (UID: \"b02fca84-f078-4d46-a842-a9b29f63eaaa\") " Oct 30 21:30:53.934256 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.934201 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-data" pod "b02fca84-f078-4d46-a842-a9b29f63eaaa" (UID: "b02fca84-f078-4d46-a842-a9b29f63eaaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:53.934465 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.934380 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-workload-socket" pod "b02fca84-f078-4d46-a842-a9b29f63eaaa" (UID: "b02fca84-f078-4d46-a842-a9b29f63eaaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:53.934679 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.934653 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02fca84-f078-4d46-a842-a9b29f63eaaa-istiod-ca-cert" pod "b02fca84-f078-4d46-a842-a9b29f63eaaa" (UID: "b02fca84-f078-4d46-a842-a9b29f63eaaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:53.934856 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.934836 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:53.934908 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.934864 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:53.934908 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.934881 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:53.934908 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.934895 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:53.935026 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.934910 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b02fca84-f078-4d46-a842-a9b29f63eaaa-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:53.936138 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.936114 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-podinfo" pod "b02fca84-f078-4d46-a842-a9b29f63eaaa" (UID: "b02fca84-f078-4d46-a842-a9b29f63eaaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:30:53.936630 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.936477 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-envoy" pod "b02fca84-f078-4d46-a842-a9b29f63eaaa" (UID: "b02fca84-f078-4d46-a842-a9b29f63eaaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:53.937043 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.937020 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-token" pod "b02fca84-f078-4d46-a842-a9b29f63eaaa" (UID: "b02fca84-f078-4d46-a842-a9b29f63eaaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:53.937150 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:53.937055 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02fca84-f078-4d46-a842-a9b29f63eaaa-kube-api-access-bjdt6" pod "b02fca84-f078-4d46-a842-a9b29f63eaaa" (UID: "b02fca84-f078-4d46-a842-a9b29f63eaaa"). InnerVolumeSpecName "kube-api-access-bjdt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:54.036878 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:54.036841 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:54.036878 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:54.036877 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjdt6\" (UniqueName: \"kubernetes.io/projected/b02fca84-f078-4d46-a842-a9b29f63eaaa-kube-api-access-bjdt6\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:54.036878 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:54.036893 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:54.037146 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:54.036908 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b02fca84-f078-4d46-a842-a9b29f63eaaa-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:30:54.533280 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:54.533247 4848 generic.go:356] "Generic (PLEG): container finished" podID="b02fca84-f078-4d46-a842-a9b29f63eaaa" containerID="4ce63e6a98d91d742876bc882581170a7fc53b000a45b9fdc6e028d47cd7d7c9" exitCode=0 Oct 30 21:30:54.533454 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:54.533299 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" Oct 30 21:30:54.533454 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:54.533324 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" event={"ID":"b02fca84-f078-4d46-a842-a9b29f63eaaa","Type":"ContainerDied","Data":"4ce63e6a98d91d742876bc882581170a7fc53b000a45b9fdc6e028d47cd7d7c9"} Oct 30 21:30:54.533454 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:54.533382 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf" event={"ID":"b02fca84-f078-4d46-a842-a9b29f63eaaa","Type":"ContainerDied","Data":"34eb69b8d466c7f2fc36ab7e374d88e9f10e5c0b8f2a77d487265c95980bc3a0"} Oct 30 21:30:54.533454 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:54.533405 4848 scope.go:122] "RemoveContainer" containerID="4ce63e6a98d91d742876bc882581170a7fc53b000a45b9fdc6e028d47cd7d7c9" Oct 30 21:30:54.544789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:54.544753 4848 scope.go:122] "RemoveContainer" containerID="0a84b750ea01dadef8d7b57baf99397ec6cfe24d43c7e17c771770418df3b565" Oct 30 21:30:54.555168 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:54.555147 4848 scope.go:122] "RemoveContainer" containerID="4ce63e6a98d91d742876bc882581170a7fc53b000a45b9fdc6e028d47cd7d7c9" Oct 30 21:30:54.555434 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:30:54.555415 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ce63e6a98d91d742876bc882581170a7fc53b000a45b9fdc6e028d47cd7d7c9\": container with ID starting with 4ce63e6a98d91d742876bc882581170a7fc53b000a45b9fdc6e028d47cd7d7c9 not found: ID does not exist" containerID="4ce63e6a98d91d742876bc882581170a7fc53b000a45b9fdc6e028d47cd7d7c9" Oct 30 21:30:54.555513 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:54.555447 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ce63e6a98d91d742876bc882581170a7fc53b000a45b9fdc6e028d47cd7d7c9"} err="failed to get container status \"4ce63e6a98d91d742876bc882581170a7fc53b000a45b9fdc6e028d47cd7d7c9\": rpc error: code = NotFound desc = could not find container \"4ce63e6a98d91d742876bc882581170a7fc53b000a45b9fdc6e028d47cd7d7c9\": container with ID starting with 4ce63e6a98d91d742876bc882581170a7fc53b000a45b9fdc6e028d47cd7d7c9 not found: ID does not exist" Oct 30 21:30:54.555513 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:54.555472 4848 scope.go:122] "RemoveContainer" containerID="0a84b750ea01dadef8d7b57baf99397ec6cfe24d43c7e17c771770418df3b565" Oct 30 21:30:54.555743 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:30:54.555724 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a84b750ea01dadef8d7b57baf99397ec6cfe24d43c7e17c771770418df3b565\": container with ID starting with 0a84b750ea01dadef8d7b57baf99397ec6cfe24d43c7e17c771770418df3b565 not found: ID does not exist" containerID="0a84b750ea01dadef8d7b57baf99397ec6cfe24d43c7e17c771770418df3b565" Oct 30 21:30:54.555830 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:54.555749 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a84b750ea01dadef8d7b57baf99397ec6cfe24d43c7e17c771770418df3b565"} err="failed to get container status \"0a84b750ea01dadef8d7b57baf99397ec6cfe24d43c7e17c771770418df3b565\": rpc error: code = NotFound desc = could not find container \"0a84b750ea01dadef8d7b57baf99397ec6cfe24d43c7e17c771770418df3b565\": container with ID starting with 0a84b750ea01dadef8d7b57baf99397ec6cfe24d43c7e17c771770418df3b565 not found: ID does not exist" Oct 30 21:30:54.588892 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:54.588866 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf"] Oct 30 21:30:54.611159 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:54.611132 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a3fb70-openshift-default-98f99985f-m9hpf"] Oct 30 21:30:54.933896 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:54.933864 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b02fca84-f078-4d46-a842-a9b29f63eaaa" path="/var/lib/kubelet/pods/b02fca84-f078-4d46-a842-a9b29f63eaaa/volumes" Oct 30 21:30:55.448606 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:55.448564 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z"] Oct 30 21:30:55.448862 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:30:55.448828 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" podUID="092690bd-ffd6-4893-96b0-025b55539fd7" containerName="istio-proxy" containerID="cri-o://e4402d746afd44d6be7b8f1a08df02ff2a8b238243c108353463b54255f4db2c" gracePeriod=30 Oct 30 21:31:00.618580 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.618548 4848 generic.go:356] "Generic (PLEG): container finished" podID="092690bd-ffd6-4893-96b0-025b55539fd7" containerID="e4402d746afd44d6be7b8f1a08df02ff2a8b238243c108353463b54255f4db2c" exitCode=0 Oct 30 21:31:00.618968 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.618624 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" event={"ID":"092690bd-ffd6-4893-96b0-025b55539fd7","Type":"ContainerDied","Data":"e4402d746afd44d6be7b8f1a08df02ff2a8b238243c108353463b54255f4db2c"} Oct 30 21:31:00.762158 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.762135 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:31:00.854012 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.853976 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-workload-certs\") pod \"092690bd-ffd6-4893-96b0-025b55539fd7\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " Oct 30 21:31:00.854012 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.854015 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/092690bd-ffd6-4893-96b0-025b55539fd7-istio-token\" (UniqueName: \"kubernetes.io/projected/092690bd-ffd6-4893-96b0-025b55539fd7-istio-token\") pod \"092690bd-ffd6-4893-96b0-025b55539fd7\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " Oct 30 21:31:00.854341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.854067 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/092690bd-ffd6-4893-96b0-025b55539fd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092690bd-ffd6-4893-96b0-025b55539fd7-istiod-ca-cert\") pod \"092690bd-ffd6-4893-96b0-025b55539fd7\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " Oct 30 21:31:00.854341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.854091 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-istio-envoy\") pod \"092690bd-ffd6-4893-96b0-025b55539fd7\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " Oct 30 21:31:00.854341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.854210 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-workload-socket\") pod \"092690bd-ffd6-4893-96b0-025b55539fd7\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " Oct 30 21:31:00.854341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.854279 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/092690bd-ffd6-4893-96b0-025b55539fd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092690bd-ffd6-4893-96b0-025b55539fd7-istio-podinfo\") pod \"092690bd-ffd6-4893-96b0-025b55539fd7\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " Oct 30 21:31:00.854341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.854301 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-workload-certs" pod "092690bd-ffd6-4893-96b0-025b55539fd7" (UID: "092690bd-ffd6-4893-96b0-025b55539fd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:00.854632 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.854369 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-credential-socket\") pod \"092690bd-ffd6-4893-96b0-025b55539fd7\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " Oct 30 21:31:00.854632 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.854394 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/092690bd-ffd6-4893-96b0-025b55539fd7-kube-api-access-bjpj6\" (UniqueName: \"kubernetes.io/projected/092690bd-ffd6-4893-96b0-025b55539fd7-kube-api-access-bjpj6\") pod \"092690bd-ffd6-4893-96b0-025b55539fd7\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " Oct 30 21:31:00.854632 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.854484 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-istio-data\") pod \"092690bd-ffd6-4893-96b0-025b55539fd7\" (UID: \"092690bd-ffd6-4893-96b0-025b55539fd7\") " Oct 30 21:31:00.854632 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.854518 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092690bd-ffd6-4893-96b0-025b55539fd7-istiod-ca-cert" pod "092690bd-ffd6-4893-96b0-025b55539fd7" (UID: "092690bd-ffd6-4893-96b0-025b55539fd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:00.854886 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.854646 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-workload-socket" pod "092690bd-ffd6-4893-96b0-025b55539fd7" (UID: "092690bd-ffd6-4893-96b0-025b55539fd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:00.854886 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.854838 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-istio-data" pod "092690bd-ffd6-4893-96b0-025b55539fd7" (UID: "092690bd-ffd6-4893-96b0-025b55539fd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:00.855168 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.855044 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-credential-socket" pod "092690bd-ffd6-4893-96b0-025b55539fd7" (UID: "092690bd-ffd6-4893-96b0-025b55539fd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:00.855304 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.855269 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:00.855304 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.855287 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:00.855304 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.855299 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092690bd-ffd6-4893-96b0-025b55539fd7-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:00.855499 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.855312 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:00.855499 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.855327 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:00.856729 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.856697 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092690bd-ffd6-4893-96b0-025b55539fd7-istio-token" pod "092690bd-ffd6-4893-96b0-025b55539fd7" (UID: "092690bd-ffd6-4893-96b0-025b55539fd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:00.856871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.856795 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-istio-envoy" pod "092690bd-ffd6-4893-96b0-025b55539fd7" (UID: "092690bd-ffd6-4893-96b0-025b55539fd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:00.857109 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.857077 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092690bd-ffd6-4893-96b0-025b55539fd7-kube-api-access-bjpj6" pod "092690bd-ffd6-4893-96b0-025b55539fd7" (UID: "092690bd-ffd6-4893-96b0-025b55539fd7"). InnerVolumeSpecName "kube-api-access-bjpj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:00.857212 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.857170 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/092690bd-ffd6-4893-96b0-025b55539fd7-istio-podinfo" pod "092690bd-ffd6-4893-96b0-025b55539fd7" (UID: "092690bd-ffd6-4893-96b0-025b55539fd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:31:00.957239 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.957207 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjpj6\" (UniqueName: \"kubernetes.io/projected/092690bd-ffd6-4893-96b0-025b55539fd7-kube-api-access-bjpj6\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:00.957239 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.957239 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092690bd-ffd6-4893-96b0-025b55539fd7-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:00.957412 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.957253 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092690bd-ffd6-4893-96b0-025b55539fd7-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:00.957412 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:00.957265 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092690bd-ffd6-4893-96b0-025b55539fd7-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:01.285883 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.285843 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w"] Oct 30 21:31:01.286104 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.286074 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" podUID="ac5c2e61-6970-43c5-9fb4-1f3991a49466" containerName="maas-api" containerID="cri-o://292d27bb2eb379c9a32140b8d71caafd0e1d42854a051676392b6f055f901584" gracePeriod=30 Oct 30 21:31:01.692082 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.688719 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" event={"ID":"092690bd-ffd6-4893-96b0-025b55539fd7","Type":"ContainerDied","Data":"775d69ab066522f7cdadffece867dcb819008440e4901107f60c629a95ee80eb"} Oct 30 21:31:01.692082 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.688837 4848 scope.go:122] "RemoveContainer" containerID="e4402d746afd44d6be7b8f1a08df02ff2a8b238243c108353463b54255f4db2c" Oct 30 21:31:01.692082 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.689007 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z" Oct 30 21:31:01.701355 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.699635 4848 generic.go:356] "Generic (PLEG): container finished" podID="ac5c2e61-6970-43c5-9fb4-1f3991a49466" containerID="292d27bb2eb379c9a32140b8d71caafd0e1d42854a051676392b6f055f901584" exitCode=0 Oct 30 21:31:01.701355 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.699716 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" event={"ID":"ac5c2e61-6970-43c5-9fb4-1f3991a49466","Type":"ContainerDied","Data":"292d27bb2eb379c9a32140b8d71caafd0e1d42854a051676392b6f055f901584"} Oct 30 21:31:01.719260 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.719228 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z"] Oct 30 21:31:01.722913 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.722889 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e1e0f5-openshift-default-dd58f9dcc-5js6z"] Oct 30 21:31:01.774411 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.774380 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" Oct 30 21:31:01.879180 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.879136 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac5c2e61-6970-43c5-9fb4-1f3991a49466-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5c2e61-6970-43c5-9fb4-1f3991a49466-openshift-service-ca\") pod \"ac5c2e61-6970-43c5-9fb4-1f3991a49466\" (UID: \"ac5c2e61-6970-43c5-9fb4-1f3991a49466\") " Oct 30 21:31:01.879383 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.879318 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac5c2e61-6970-43c5-9fb4-1f3991a49466-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5c2e61-6970-43c5-9fb4-1f3991a49466-metrics-certs\") pod \"ac5c2e61-6970-43c5-9fb4-1f3991a49466\" (UID: \"ac5c2e61-6970-43c5-9fb4-1f3991a49466\") " Oct 30 21:31:01.879538 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.879512 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5c2e61-6970-43c5-9fb4-1f3991a49466-openshift-service-ca" pod "ac5c2e61-6970-43c5-9fb4-1f3991a49466" (UID: "ac5c2e61-6970-43c5-9fb4-1f3991a49466"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:01.879598 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.879567 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac5c2e61-6970-43c5-9fb4-1f3991a49466-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5c2e61-6970-43c5-9fb4-1f3991a49466-maas-api-tls\") pod \"ac5c2e61-6970-43c5-9fb4-1f3991a49466\" (UID: \"ac5c2e61-6970-43c5-9fb4-1f3991a49466\") " Oct 30 21:31:01.879645 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.879602 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac5c2e61-6970-43c5-9fb4-1f3991a49466-kube-api-access-bq6pc\" (UniqueName: \"kubernetes.io/projected/ac5c2e61-6970-43c5-9fb4-1f3991a49466-kube-api-access-bq6pc\") pod \"ac5c2e61-6970-43c5-9fb4-1f3991a49466\" (UID: \"ac5c2e61-6970-43c5-9fb4-1f3991a49466\") " Oct 30 21:31:01.880272 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.880247 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5c2e61-6970-43c5-9fb4-1f3991a49466-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:01.882456 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.882432 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5c2e61-6970-43c5-9fb4-1f3991a49466-kube-api-access-bq6pc" pod "ac5c2e61-6970-43c5-9fb4-1f3991a49466" (UID: "ac5c2e61-6970-43c5-9fb4-1f3991a49466"). InnerVolumeSpecName "kube-api-access-bq6pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:01.882559 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.882455 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5c2e61-6970-43c5-9fb4-1f3991a49466-metrics-certs" pod "ac5c2e61-6970-43c5-9fb4-1f3991a49466" (UID: "ac5c2e61-6970-43c5-9fb4-1f3991a49466"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:01.882559 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.882471 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5c2e61-6970-43c5-9fb4-1f3991a49466-maas-api-tls" pod "ac5c2e61-6970-43c5-9fb4-1f3991a49466" (UID: "ac5c2e61-6970-43c5-9fb4-1f3991a49466"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:01.982818 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.982712 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac5c2e61-6970-43c5-9fb4-1f3991a49466-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:01.982818 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.982749 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac5c2e61-6970-43c5-9fb4-1f3991a49466-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:01.982818 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:01.982779 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq6pc\" (UniqueName: \"kubernetes.io/projected/ac5c2e61-6970-43c5-9fb4-1f3991a49466-kube-api-access-bq6pc\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:02.711069 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:02.711024 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" event={"ID":"ac5c2e61-6970-43c5-9fb4-1f3991a49466","Type":"ContainerDied","Data":"f501c624e9486ddbc4273646ba3fe7b68bd0cf0b87601ecb0c75b08b9ecf3dba"} Oct 30 21:31:02.711618 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:02.711080 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w" Oct 30 21:31:02.711618 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:02.711117 4848 scope.go:122] "RemoveContainer" containerID="292d27bb2eb379c9a32140b8d71caafd0e1d42854a051676392b6f055f901584" Oct 30 21:31:02.747432 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:02.747403 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w"] Oct 30 21:31:02.751953 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:02.751922 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e8063c-858c5c7d87-cmd5w"] Oct 30 21:31:02.932842 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:02.932810 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="092690bd-ffd6-4893-96b0-025b55539fd7" path="/var/lib/kubelet/pods/092690bd-ffd6-4893-96b0-025b55539fd7/volumes" Oct 30 21:31:02.933311 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:02.933296 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac5c2e61-6970-43c5-9fb4-1f3991a49466" path="/var/lib/kubelet/pods/ac5c2e61-6970-43c5-9fb4-1f3991a49466/volumes" Oct 30 21:31:05.843060 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:05.843025 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq"] Oct 30 21:31:05.843587 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:05.843273 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" podUID="813afec3-2ef0-434f-b4bd-08c9c705c6ec" containerName="maas-api" containerID="cri-o://ce9698fee3857a1172705c68432d57ff35079040aa20506e488792140aa175a1" gracePeriod=30 Oct 30 21:31:06.321638 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.321611 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" Oct 30 21:31:06.380708 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.380678 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/813afec3-2ef0-434f-b4bd-08c9c705c6ec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/813afec3-2ef0-434f-b4bd-08c9c705c6ec-metrics-certs\") pod \"813afec3-2ef0-434f-b4bd-08c9c705c6ec\" (UID: \"813afec3-2ef0-434f-b4bd-08c9c705c6ec\") " Oct 30 21:31:06.380927 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.380840 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/813afec3-2ef0-434f-b4bd-08c9c705c6ec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/813afec3-2ef0-434f-b4bd-08c9c705c6ec-openshift-service-ca\") pod \"813afec3-2ef0-434f-b4bd-08c9c705c6ec\" (UID: \"813afec3-2ef0-434f-b4bd-08c9c705c6ec\") " Oct 30 21:31:06.380927 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.380898 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/813afec3-2ef0-434f-b4bd-08c9c705c6ec-kube-api-access-l2d7d\" (UniqueName: \"kubernetes.io/projected/813afec3-2ef0-434f-b4bd-08c9c705c6ec-kube-api-access-l2d7d\") pod \"813afec3-2ef0-434f-b4bd-08c9c705c6ec\" (UID: \"813afec3-2ef0-434f-b4bd-08c9c705c6ec\") " Oct 30 21:31:06.380927 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.380926 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/813afec3-2ef0-434f-b4bd-08c9c705c6ec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/813afec3-2ef0-434f-b4bd-08c9c705c6ec-maas-api-tls\") pod \"813afec3-2ef0-434f-b4bd-08c9c705c6ec\" (UID: \"813afec3-2ef0-434f-b4bd-08c9c705c6ec\") " Oct 30 21:31:06.381326 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.381304 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/813afec3-2ef0-434f-b4bd-08c9c705c6ec-openshift-service-ca" pod "813afec3-2ef0-434f-b4bd-08c9c705c6ec" (UID: "813afec3-2ef0-434f-b4bd-08c9c705c6ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:06.381510 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.381493 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/813afec3-2ef0-434f-b4bd-08c9c705c6ec-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:06.383259 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.383231 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/813afec3-2ef0-434f-b4bd-08c9c705c6ec-maas-api-tls" pod "813afec3-2ef0-434f-b4bd-08c9c705c6ec" (UID: "813afec3-2ef0-434f-b4bd-08c9c705c6ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:06.383361 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.383332 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/813afec3-2ef0-434f-b4bd-08c9c705c6ec-metrics-certs" pod "813afec3-2ef0-434f-b4bd-08c9c705c6ec" (UID: "813afec3-2ef0-434f-b4bd-08c9c705c6ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:06.383361 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.383333 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813afec3-2ef0-434f-b4bd-08c9c705c6ec-kube-api-access-l2d7d" pod "813afec3-2ef0-434f-b4bd-08c9c705c6ec" (UID: "813afec3-2ef0-434f-b4bd-08c9c705c6ec"). InnerVolumeSpecName "kube-api-access-l2d7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:06.483787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.483116 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/813afec3-2ef0-434f-b4bd-08c9c705c6ec-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:06.483787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.483152 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2d7d\" (UniqueName: \"kubernetes.io/projected/813afec3-2ef0-434f-b4bd-08c9c705c6ec-kube-api-access-l2d7d\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:06.483787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.483172 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/813afec3-2ef0-434f-b4bd-08c9c705c6ec-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:06.742387 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.742349 4848 generic.go:356] "Generic (PLEG): container finished" podID="813afec3-2ef0-434f-b4bd-08c9c705c6ec" containerID="ce9698fee3857a1172705c68432d57ff35079040aa20506e488792140aa175a1" exitCode=0 Oct 30 21:31:06.742584 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.742390 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" Oct 30 21:31:06.742584 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.742431 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" event={"ID":"813afec3-2ef0-434f-b4bd-08c9c705c6ec","Type":"ContainerDied","Data":"ce9698fee3857a1172705c68432d57ff35079040aa20506e488792140aa175a1"} Oct 30 21:31:06.742584 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.742496 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq" event={"ID":"813afec3-2ef0-434f-b4bd-08c9c705c6ec","Type":"ContainerDied","Data":"02647024efeae4bf4bfc0ad39f3c483588d173b4874ab3abd3ae84e662d2ddf2"} Oct 30 21:31:06.742584 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.742521 4848 scope.go:122] "RemoveContainer" containerID="ce9698fee3857a1172705c68432d57ff35079040aa20506e488792140aa175a1" Oct 30 21:31:06.753711 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.753690 4848 scope.go:122] "RemoveContainer" containerID="ce9698fee3857a1172705c68432d57ff35079040aa20506e488792140aa175a1" Oct 30 21:31:06.754025 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:31:06.754003 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce9698fee3857a1172705c68432d57ff35079040aa20506e488792140aa175a1\": container with ID starting with ce9698fee3857a1172705c68432d57ff35079040aa20506e488792140aa175a1 not found: ID does not exist" containerID="ce9698fee3857a1172705c68432d57ff35079040aa20506e488792140aa175a1" Oct 30 21:31:06.754099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.754036 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce9698fee3857a1172705c68432d57ff35079040aa20506e488792140aa175a1"} err="failed to get container status \"ce9698fee3857a1172705c68432d57ff35079040aa20506e488792140aa175a1\": rpc error: code = NotFound desc = could not find container \"ce9698fee3857a1172705c68432d57ff35079040aa20506e488792140aa175a1\": container with ID starting with ce9698fee3857a1172705c68432d57ff35079040aa20506e488792140aa175a1 not found: ID does not exist" Oct 30 21:31:06.771148 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.771105 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq"] Oct 30 21:31:06.774200 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.774178 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-99668e-77c7f9f74c-mx8pq"] Oct 30 21:31:06.933876 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:06.933841 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="813afec3-2ef0-434f-b4bd-08c9c705c6ec" path="/var/lib/kubelet/pods/813afec3-2ef0-434f-b4bd-08c9c705c6ec/volumes" Oct 30 21:31:07.075100 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.075012 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf"] Oct 30 21:31:07.075310 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.075282 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" podUID="fd7a438e-6705-4f55-8580-fafb7a21f4f0" containerName="maas-api" containerID="cri-o://a5e567571038c54ede10760341e7a856808dd38f65ad95616a7adc1850ebcdfd" gracePeriod=30 Oct 30 21:31:07.559546 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.548346 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" Oct 30 21:31:07.678480 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.678445 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd7a438e-6705-4f55-8580-fafb7a21f4f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd7a438e-6705-4f55-8580-fafb7a21f4f0-maas-api-tls\") pod \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\" (UID: \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\") " Oct 30 21:31:07.678679 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.678551 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7a438e-6705-4f55-8580-fafb7a21f4f0-kube-api-access-qcjbb\" (UniqueName: \"kubernetes.io/projected/fd7a438e-6705-4f55-8580-fafb7a21f4f0-kube-api-access-qcjbb\") pod \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\" (UID: \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\") " Oct 30 21:31:07.678679 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.678633 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd7a438e-6705-4f55-8580-fafb7a21f4f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd7a438e-6705-4f55-8580-fafb7a21f4f0-openshift-service-ca\") pod \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\" (UID: \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\") " Oct 30 21:31:07.679117 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.679087 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7a438e-6705-4f55-8580-fafb7a21f4f0-openshift-service-ca" pod "fd7a438e-6705-4f55-8580-fafb7a21f4f0" (UID: "fd7a438e-6705-4f55-8580-fafb7a21f4f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:07.679214 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.679094 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd7a438e-6705-4f55-8580-fafb7a21f4f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd7a438e-6705-4f55-8580-fafb7a21f4f0-metrics-certs\") pod \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\" (UID: \"fd7a438e-6705-4f55-8580-fafb7a21f4f0\") " Oct 30 21:31:07.679830 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.679804 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd7a438e-6705-4f55-8580-fafb7a21f4f0-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:07.680992 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.680964 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7a438e-6705-4f55-8580-fafb7a21f4f0-kube-api-access-qcjbb" pod "fd7a438e-6705-4f55-8580-fafb7a21f4f0" (UID: "fd7a438e-6705-4f55-8580-fafb7a21f4f0"). InnerVolumeSpecName "kube-api-access-qcjbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:07.681082 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.681007 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd7a438e-6705-4f55-8580-fafb7a21f4f0-maas-api-tls" pod "fd7a438e-6705-4f55-8580-fafb7a21f4f0" (UID: "fd7a438e-6705-4f55-8580-fafb7a21f4f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:07.681082 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.681035 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd7a438e-6705-4f55-8580-fafb7a21f4f0-metrics-certs" pod "fd7a438e-6705-4f55-8580-fafb7a21f4f0" (UID: "fd7a438e-6705-4f55-8580-fafb7a21f4f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:07.755353 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.755254 4848 generic.go:356] "Generic (PLEG): container finished" podID="fd7a438e-6705-4f55-8580-fafb7a21f4f0" containerID="a5e567571038c54ede10760341e7a856808dd38f65ad95616a7adc1850ebcdfd" exitCode=0 Oct 30 21:31:07.755353 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.755326 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" Oct 30 21:31:07.755579 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.755500 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" event={"ID":"fd7a438e-6705-4f55-8580-fafb7a21f4f0","Type":"ContainerDied","Data":"a5e567571038c54ede10760341e7a856808dd38f65ad95616a7adc1850ebcdfd"} Oct 30 21:31:07.755579 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.755552 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf" event={"ID":"fd7a438e-6705-4f55-8580-fafb7a21f4f0","Type":"ContainerDied","Data":"e0e8bcc35e4e04e770bcf8e8164fe378a2d819f151d5796d53d355638b7b7c64"} Oct 30 21:31:07.755714 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.755597 4848 scope.go:122] "RemoveContainer" containerID="a5e567571038c54ede10760341e7a856808dd38f65ad95616a7adc1850ebcdfd" Oct 30 21:31:07.767959 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.767939 4848 scope.go:122] "RemoveContainer" containerID="a5e567571038c54ede10760341e7a856808dd38f65ad95616a7adc1850ebcdfd" Oct 30 21:31:07.768271 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:31:07.768248 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e567571038c54ede10760341e7a856808dd38f65ad95616a7adc1850ebcdfd\": container with ID starting with a5e567571038c54ede10760341e7a856808dd38f65ad95616a7adc1850ebcdfd not found: ID does not exist" containerID="a5e567571038c54ede10760341e7a856808dd38f65ad95616a7adc1850ebcdfd" Oct 30 21:31:07.768350 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.768279 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e567571038c54ede10760341e7a856808dd38f65ad95616a7adc1850ebcdfd"} err="failed to get container status \"a5e567571038c54ede10760341e7a856808dd38f65ad95616a7adc1850ebcdfd\": rpc error: code = NotFound desc = could not find container \"a5e567571038c54ede10760341e7a856808dd38f65ad95616a7adc1850ebcdfd\": container with ID starting with a5e567571038c54ede10760341e7a856808dd38f65ad95616a7adc1850ebcdfd not found: ID does not exist" Oct 30 21:31:07.781145 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.781108 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf"] Oct 30 21:31:07.781417 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.781393 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd7a438e-6705-4f55-8580-fafb7a21f4f0-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:07.781461 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.781423 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd7a438e-6705-4f55-8580-fafb7a21f4f0-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:07.781461 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.781433 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcjbb\" (UniqueName: \"kubernetes.io/projected/fd7a438e-6705-4f55-8580-fafb7a21f4f0-kube-api-access-qcjbb\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:07.784577 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:07.784552 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5506f4-54b7765c87-rxdtf"] Oct 30 21:31:08.933393 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:08.933359 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd7a438e-6705-4f55-8580-fafb7a21f4f0" path="/var/lib/kubelet/pods/fd7a438e-6705-4f55-8580-fafb7a21f4f0/volumes" Oct 30 21:31:10.036652 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:10.036620 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq"] Oct 30 21:31:10.037184 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:10.036911 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" podUID="11a4ae21-6152-44d6-9327-18c9a856261c" containerName="istio-proxy" containerID="cri-o://1588523f2f7b16543e4b924f7b5b5e26b2f8420a4a5d6e4aa331c71882155d73" gracePeriod=30 Oct 30 21:31:13.824686 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:13.824490 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd"] Oct 30 21:31:13.825235 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:13.824747 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" podUID="127e9b91-a76e-46f6-9970-e5dcf8a0ba78" containerName="istio-proxy" containerID="cri-o://15c1a27811b63dc15f776b703c45bd27ea9179445a0f1fd504225f6dc8ed51e7" gracePeriod=30 Oct 30 21:31:14.627009 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:14.626974 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv"] Oct 30 21:31:14.627270 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:14.627189 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" podUID="c31751bb-34fe-4e1f-8bcf-2f9b457d3379" containerName="istio-proxy" containerID="cri-o://d92f44dd44cd4143c49b99d280d55b31616f254101690807bd13b10a878f58d2" gracePeriod=30 Oct 30 21:31:15.342221 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.342193 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:31:15.485482 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.485453 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a4ae21-6152-44d6-9327-18c9a856261c-kube-api-access-wfvvm\" (UniqueName: \"kubernetes.io/projected/11a4ae21-6152-44d6-9327-18c9a856261c-kube-api-access-wfvvm\") pod \"11a4ae21-6152-44d6-9327-18c9a856261c\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " Oct 30 21:31:15.485685 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.485575 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-workload-socket\") pod \"11a4ae21-6152-44d6-9327-18c9a856261c\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " Oct 30 21:31:15.485685 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.485630 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11a4ae21-6152-44d6-9327-18c9a856261c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11a4ae21-6152-44d6-9327-18c9a856261c-istio-podinfo\") pod \"11a4ae21-6152-44d6-9327-18c9a856261c\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " Oct 30 21:31:15.485916 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.485685 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-istio-data\") pod \"11a4ae21-6152-44d6-9327-18c9a856261c\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " Oct 30 21:31:15.485916 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.485819 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-credential-socket\") pod \"11a4ae21-6152-44d6-9327-18c9a856261c\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " Oct 30 21:31:15.485916 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.485867 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11a4ae21-6152-44d6-9327-18c9a856261c-istio-token\" (UniqueName: \"kubernetes.io/projected/11a4ae21-6152-44d6-9327-18c9a856261c-istio-token\") pod \"11a4ae21-6152-44d6-9327-18c9a856261c\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " Oct 30 21:31:15.486100 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.485919 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-workload-certs\") pod \"11a4ae21-6152-44d6-9327-18c9a856261c\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " Oct 30 21:31:15.486100 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.485932 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-workload-socket" pod "11a4ae21-6152-44d6-9327-18c9a856261c" (UID: "11a4ae21-6152-44d6-9327-18c9a856261c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:15.486100 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.485978 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11a4ae21-6152-44d6-9327-18c9a856261c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11a4ae21-6152-44d6-9327-18c9a856261c-istiod-ca-cert\") pod \"11a4ae21-6152-44d6-9327-18c9a856261c\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " Oct 30 21:31:15.486100 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.485974 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-istio-data" pod "11a4ae21-6152-44d6-9327-18c9a856261c" (UID: "11a4ae21-6152-44d6-9327-18c9a856261c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:15.486100 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.486011 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-istio-envoy\") pod \"11a4ae21-6152-44d6-9327-18c9a856261c\" (UID: \"11a4ae21-6152-44d6-9327-18c9a856261c\") " Oct 30 21:31:15.486100 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.486086 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-credential-socket" pod "11a4ae21-6152-44d6-9327-18c9a856261c" (UID: "11a4ae21-6152-44d6-9327-18c9a856261c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:15.486428 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.486205 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-workload-certs" pod "11a4ae21-6152-44d6-9327-18c9a856261c" (UID: "11a4ae21-6152-44d6-9327-18c9a856261c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:15.486428 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.486332 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a4ae21-6152-44d6-9327-18c9a856261c-istiod-ca-cert" pod "11a4ae21-6152-44d6-9327-18c9a856261c" (UID: "11a4ae21-6152-44d6-9327-18c9a856261c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:15.486648 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.486624 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:15.486714 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.486654 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:15.486714 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.486669 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:15.486714 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.486685 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:15.486714 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.486699 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11a4ae21-6152-44d6-9327-18c9a856261c-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:15.488047 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.488021 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11a4ae21-6152-44d6-9327-18c9a856261c-istio-podinfo" pod "11a4ae21-6152-44d6-9327-18c9a856261c" (UID: "11a4ae21-6152-44d6-9327-18c9a856261c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:31:15.488203 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.488178 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-istio-envoy" pod "11a4ae21-6152-44d6-9327-18c9a856261c" (UID: "11a4ae21-6152-44d6-9327-18c9a856261c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:15.488512 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.488486 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a4ae21-6152-44d6-9327-18c9a856261c-kube-api-access-wfvvm" pod "11a4ae21-6152-44d6-9327-18c9a856261c" (UID: "11a4ae21-6152-44d6-9327-18c9a856261c"). InnerVolumeSpecName "kube-api-access-wfvvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:15.488584 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.488527 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a4ae21-6152-44d6-9327-18c9a856261c-istio-token" pod "11a4ae21-6152-44d6-9327-18c9a856261c" (UID: "11a4ae21-6152-44d6-9327-18c9a856261c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:15.588147 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.588065 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11a4ae21-6152-44d6-9327-18c9a856261c-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:15.588147 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.588094 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11a4ae21-6152-44d6-9327-18c9a856261c-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:15.588147 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.588106 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfvvm\" (UniqueName: \"kubernetes.io/projected/11a4ae21-6152-44d6-9327-18c9a856261c-kube-api-access-wfvvm\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:15.588147 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.588116 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11a4ae21-6152-44d6-9327-18c9a856261c-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:15.834541 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.834510 4848 generic.go:356] "Generic (PLEG): container finished" podID="11a4ae21-6152-44d6-9327-18c9a856261c" containerID="1588523f2f7b16543e4b924f7b5b5e26b2f8420a4a5d6e4aa331c71882155d73" exitCode=0 Oct 30 21:31:15.834747 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.834561 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" Oct 30 21:31:15.834747 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.834590 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" event={"ID":"11a4ae21-6152-44d6-9327-18c9a856261c","Type":"ContainerDied","Data":"1588523f2f7b16543e4b924f7b5b5e26b2f8420a4a5d6e4aa331c71882155d73"} Oct 30 21:31:15.834747 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.834650 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq" event={"ID":"11a4ae21-6152-44d6-9327-18c9a856261c","Type":"ContainerDied","Data":"6a403d9b7fcd696c9aa920c35e341c3358b08a1d3deb9519d6e8730fb3220742"} Oct 30 21:31:15.834747 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.834682 4848 scope.go:122] "RemoveContainer" containerID="1588523f2f7b16543e4b924f7b5b5e26b2f8420a4a5d6e4aa331c71882155d73" Oct 30 21:31:15.846670 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.846641 4848 scope.go:122] "RemoveContainer" containerID="59bbafcc5ce9a738b54f45510a81224e29f30ac77a2609f8700b6f1190dd1a82" Oct 30 21:31:15.856538 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.856516 4848 scope.go:122] "RemoveContainer" containerID="1588523f2f7b16543e4b924f7b5b5e26b2f8420a4a5d6e4aa331c71882155d73" Oct 30 21:31:15.857000 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:31:15.856979 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1588523f2f7b16543e4b924f7b5b5e26b2f8420a4a5d6e4aa331c71882155d73\": container with ID starting with 1588523f2f7b16543e4b924f7b5b5e26b2f8420a4a5d6e4aa331c71882155d73 not found: ID does not exist" containerID="1588523f2f7b16543e4b924f7b5b5e26b2f8420a4a5d6e4aa331c71882155d73" Oct 30 21:31:15.857093 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.857008 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1588523f2f7b16543e4b924f7b5b5e26b2f8420a4a5d6e4aa331c71882155d73"} err="failed to get container status \"1588523f2f7b16543e4b924f7b5b5e26b2f8420a4a5d6e4aa331c71882155d73\": rpc error: code = NotFound desc = could not find container \"1588523f2f7b16543e4b924f7b5b5e26b2f8420a4a5d6e4aa331c71882155d73\": container with ID starting with 1588523f2f7b16543e4b924f7b5b5e26b2f8420a4a5d6e4aa331c71882155d73 not found: ID does not exist" Oct 30 21:31:15.857093 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.857025 4848 scope.go:122] "RemoveContainer" containerID="59bbafcc5ce9a738b54f45510a81224e29f30ac77a2609f8700b6f1190dd1a82" Oct 30 21:31:15.857320 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:31:15.857297 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59bbafcc5ce9a738b54f45510a81224e29f30ac77a2609f8700b6f1190dd1a82\": container with ID starting with 59bbafcc5ce9a738b54f45510a81224e29f30ac77a2609f8700b6f1190dd1a82 not found: ID does not exist" containerID="59bbafcc5ce9a738b54f45510a81224e29f30ac77a2609f8700b6f1190dd1a82" Oct 30 21:31:15.857363 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.857332 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59bbafcc5ce9a738b54f45510a81224e29f30ac77a2609f8700b6f1190dd1a82"} err="failed to get container status \"59bbafcc5ce9a738b54f45510a81224e29f30ac77a2609f8700b6f1190dd1a82\": rpc error: code = NotFound desc = could not find container \"59bbafcc5ce9a738b54f45510a81224e29f30ac77a2609f8700b6f1190dd1a82\": container with ID starting with 59bbafcc5ce9a738b54f45510a81224e29f30ac77a2609f8700b6f1190dd1a82 not found: ID does not exist" Oct 30 21:31:15.859203 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.859182 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq"] Oct 30 21:31:15.867073 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:15.867047 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e8063c-openshift-default-dc6f7fbc7-m66rq"] Oct 30 21:31:16.851761 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:16.851726 4848 patch_prober.go:29] interesting pod/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:31:16.852176 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:16.851793 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" podUID="c31751bb-34fe-4e1f-8bcf-2f9b457d3379" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:31:16.933481 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:16.933446 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11a4ae21-6152-44d6-9327-18c9a856261c" path="/var/lib/kubelet/pods/11a4ae21-6152-44d6-9327-18c9a856261c/volumes" Oct 30 21:31:17.408491 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:17.408450 4848 patch_prober.go:29] interesting pod/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:31:17.408673 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:17.408509 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" podUID="127e9b91-a76e-46f6-9970-e5dcf8a0ba78" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:31:19.122254 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.122228 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:31:19.238185 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.238150 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-workload-certs\") pod \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " Oct 30 21:31:19.238382 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.238197 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-token\" (UniqueName: \"kubernetes.io/projected/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-token\") pod \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " Oct 30 21:31:19.238382 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.238228 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istiod-ca-cert\") pod \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " Oct 30 21:31:19.238382 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.238270 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-podinfo\") pod \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " Oct 30 21:31:19.238382 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.238300 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-envoy\") pod \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " Oct 30 21:31:19.238382 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.238330 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-credential-socket\") pod \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " Oct 30 21:31:19.238669 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.238423 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-data\") pod \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " Oct 30 21:31:19.238669 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.238479 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-workload-certs" pod "127e9b91-a76e-46f6-9970-e5dcf8a0ba78" (UID: "127e9b91-a76e-46f6-9970-e5dcf8a0ba78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:19.238669 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.238519 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-workload-socket\") pod \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " Oct 30 21:31:19.238669 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.238631 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-kube-api-access-795m2\" (UniqueName: \"kubernetes.io/projected/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-kube-api-access-795m2\") pod \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\" (UID: \"127e9b91-a76e-46f6-9970-e5dcf8a0ba78\") " Oct 30 21:31:19.238925 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.238745 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-credential-socket" pod "127e9b91-a76e-46f6-9970-e5dcf8a0ba78" (UID: "127e9b91-a76e-46f6-9970-e5dcf8a0ba78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:19.238925 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.238758 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-data" pod "127e9b91-a76e-46f6-9970-e5dcf8a0ba78" (UID: "127e9b91-a76e-46f6-9970-e5dcf8a0ba78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:19.238925 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.238826 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istiod-ca-cert" pod "127e9b91-a76e-46f6-9970-e5dcf8a0ba78" (UID: "127e9b91-a76e-46f6-9970-e5dcf8a0ba78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:19.239242 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.239112 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-workload-socket" pod "127e9b91-a76e-46f6-9970-e5dcf8a0ba78" (UID: "127e9b91-a76e-46f6-9970-e5dcf8a0ba78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:19.239378 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.239311 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:19.239378 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.239333 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:19.239378 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.239349 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:19.239378 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.239367 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:19.239628 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.239382 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:19.240659 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.240636 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-podinfo" pod "127e9b91-a76e-46f6-9970-e5dcf8a0ba78" (UID: "127e9b91-a76e-46f6-9970-e5dcf8a0ba78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:31:19.240659 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.240640 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-token" pod "127e9b91-a76e-46f6-9970-e5dcf8a0ba78" (UID: "127e9b91-a76e-46f6-9970-e5dcf8a0ba78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:19.240909 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.240891 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-kube-api-access-795m2" pod "127e9b91-a76e-46f6-9970-e5dcf8a0ba78" (UID: "127e9b91-a76e-46f6-9970-e5dcf8a0ba78"). InnerVolumeSpecName "kube-api-access-795m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:19.240950 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.240929 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-envoy" pod "127e9b91-a76e-46f6-9970-e5dcf8a0ba78" (UID: "127e9b91-a76e-46f6-9970-e5dcf8a0ba78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:19.341468 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.341422 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:19.341674 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.341476 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:19.341674 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.341493 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:19.341674 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.341509 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-795m2\" (UniqueName: \"kubernetes.io/projected/127e9b91-a76e-46f6-9970-e5dcf8a0ba78-kube-api-access-795m2\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:19.874591 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.869889 4848 generic.go:356] "Generic (PLEG): container finished" podID="127e9b91-a76e-46f6-9970-e5dcf8a0ba78" containerID="15c1a27811b63dc15f776b703c45bd27ea9179445a0f1fd504225f6dc8ed51e7" exitCode=0 Oct 30 21:31:19.874591 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.870039 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" event={"ID":"127e9b91-a76e-46f6-9970-e5dcf8a0ba78","Type":"ContainerDied","Data":"15c1a27811b63dc15f776b703c45bd27ea9179445a0f1fd504225f6dc8ed51e7"} Oct 30 21:31:19.874591 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.870094 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" event={"ID":"127e9b91-a76e-46f6-9970-e5dcf8a0ba78","Type":"ContainerDied","Data":"cde5c62430111eac3bc5ad35dfb2673c2d95a91ba28288acf446eaced4860d96"} Oct 30 21:31:19.874591 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.870125 4848 scope.go:122] "RemoveContainer" containerID="15c1a27811b63dc15f776b703c45bd27ea9179445a0f1fd504225f6dc8ed51e7" Oct 30 21:31:19.874591 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.870279 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd" Oct 30 21:31:19.883040 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.878685 4848 generic.go:356] "Generic (PLEG): container finished" podID="c31751bb-34fe-4e1f-8bcf-2f9b457d3379" containerID="d92f44dd44cd4143c49b99d280d55b31616f254101690807bd13b10a878f58d2" exitCode=0 Oct 30 21:31:19.883040 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.878886 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" event={"ID":"c31751bb-34fe-4e1f-8bcf-2f9b457d3379","Type":"ContainerDied","Data":"d92f44dd44cd4143c49b99d280d55b31616f254101690807bd13b10a878f58d2"} Oct 30 21:31:19.890326 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.890308 4848 scope.go:122] "RemoveContainer" containerID="623d607cb97ef2790866d8baa631053d6a9e8f0400dd5e4ac0d59e36cc83849f" Oct 30 21:31:19.902550 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.902487 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd"] Oct 30 21:31:19.909035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.909006 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-99668e-openshift-default-67c7577f5c-gq2xd"] Oct 30 21:31:19.911988 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.911970 4848 scope.go:122] "RemoveContainer" containerID="15c1a27811b63dc15f776b703c45bd27ea9179445a0f1fd504225f6dc8ed51e7" Oct 30 21:31:19.912326 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:31:19.912298 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c1a27811b63dc15f776b703c45bd27ea9179445a0f1fd504225f6dc8ed51e7\": container with ID starting with 15c1a27811b63dc15f776b703c45bd27ea9179445a0f1fd504225f6dc8ed51e7 not found: ID does not exist" containerID="15c1a27811b63dc15f776b703c45bd27ea9179445a0f1fd504225f6dc8ed51e7" Oct 30 21:31:19.912389 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.912346 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c1a27811b63dc15f776b703c45bd27ea9179445a0f1fd504225f6dc8ed51e7"} err="failed to get container status \"15c1a27811b63dc15f776b703c45bd27ea9179445a0f1fd504225f6dc8ed51e7\": rpc error: code = NotFound desc = could not find container \"15c1a27811b63dc15f776b703c45bd27ea9179445a0f1fd504225f6dc8ed51e7\": container with ID starting with 15c1a27811b63dc15f776b703c45bd27ea9179445a0f1fd504225f6dc8ed51e7 not found: ID does not exist" Oct 30 21:31:19.912389 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.912373 4848 scope.go:122] "RemoveContainer" containerID="623d607cb97ef2790866d8baa631053d6a9e8f0400dd5e4ac0d59e36cc83849f" Oct 30 21:31:19.912666 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:31:19.912649 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"623d607cb97ef2790866d8baa631053d6a9e8f0400dd5e4ac0d59e36cc83849f\": container with ID starting with 623d607cb97ef2790866d8baa631053d6a9e8f0400dd5e4ac0d59e36cc83849f not found: ID does not exist" containerID="623d607cb97ef2790866d8baa631053d6a9e8f0400dd5e4ac0d59e36cc83849f" Oct 30 21:31:19.912721 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.912676 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623d607cb97ef2790866d8baa631053d6a9e8f0400dd5e4ac0d59e36cc83849f"} err="failed to get container status \"623d607cb97ef2790866d8baa631053d6a9e8f0400dd5e4ac0d59e36cc83849f\": rpc error: code = NotFound desc = could not find container \"623d607cb97ef2790866d8baa631053d6a9e8f0400dd5e4ac0d59e36cc83849f\": container with ID starting with 623d607cb97ef2790866d8baa631053d6a9e8f0400dd5e4ac0d59e36cc83849f not found: ID does not exist" Oct 30 21:31:19.930505 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:19.930479 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:31:20.051016 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.050978 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-workload-socket\") pod \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " Oct 30 21:31:20.051205 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.051086 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istiod-ca-cert\") pod \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " Oct 30 21:31:20.051205 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.051132 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-credential-socket\") pod \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " Oct 30 21:31:20.051205 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.051186 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-envoy\") pod \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " Oct 30 21:31:20.051371 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.051263 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-data\") pod \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " Oct 30 21:31:20.051371 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.051326 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-kube-api-access-2smmh\" (UniqueName: \"kubernetes.io/projected/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-kube-api-access-2smmh\") pod \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " Oct 30 21:31:20.051371 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.051322 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-workload-socket" pod "c31751bb-34fe-4e1f-8bcf-2f9b457d3379" (UID: "c31751bb-34fe-4e1f-8bcf-2f9b457d3379"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:20.051542 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.051401 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-token\" (UniqueName: \"kubernetes.io/projected/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-token\") pod \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " Oct 30 21:31:20.051542 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.051432 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-workload-certs\") pod \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " Oct 30 21:31:20.051542 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.051433 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-credential-socket" pod "c31751bb-34fe-4e1f-8bcf-2f9b457d3379" (UID: "c31751bb-34fe-4e1f-8bcf-2f9b457d3379"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:20.051542 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.051518 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-podinfo\") pod \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\" (UID: \"c31751bb-34fe-4e1f-8bcf-2f9b457d3379\") " Oct 30 21:31:20.051542 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.051532 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istiod-ca-cert" pod "c31751bb-34fe-4e1f-8bcf-2f9b457d3379" (UID: "c31751bb-34fe-4e1f-8bcf-2f9b457d3379"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:20.051877 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.051574 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-data" pod "c31751bb-34fe-4e1f-8bcf-2f9b457d3379" (UID: "c31751bb-34fe-4e1f-8bcf-2f9b457d3379"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:20.051877 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.051719 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-workload-certs" pod "c31751bb-34fe-4e1f-8bcf-2f9b457d3379" (UID: "c31751bb-34fe-4e1f-8bcf-2f9b457d3379"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:20.052299 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.052275 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:20.052299 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.052302 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:20.052508 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.052321 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:20.052508 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.052345 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:20.052508 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.052359 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:20.053911 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.053877 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-envoy" pod "c31751bb-34fe-4e1f-8bcf-2f9b457d3379" (UID: "c31751bb-34fe-4e1f-8bcf-2f9b457d3379"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:20.054039 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.054019 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-kube-api-access-2smmh" pod "c31751bb-34fe-4e1f-8bcf-2f9b457d3379" (UID: "c31751bb-34fe-4e1f-8bcf-2f9b457d3379"). InnerVolumeSpecName "kube-api-access-2smmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:20.054099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.054044 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-token" pod "c31751bb-34fe-4e1f-8bcf-2f9b457d3379" (UID: "c31751bb-34fe-4e1f-8bcf-2f9b457d3379"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:20.054194 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.054180 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-podinfo" pod "c31751bb-34fe-4e1f-8bcf-2f9b457d3379" (UID: "c31751bb-34fe-4e1f-8bcf-2f9b457d3379"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:31:20.154298 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.154209 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:20.154298 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.154243 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2smmh\" (UniqueName: \"kubernetes.io/projected/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-kube-api-access-2smmh\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:20.154298 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.154255 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:20.154298 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.154264 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c31751bb-34fe-4e1f-8bcf-2f9b457d3379-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:20.886726 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.886697 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" Oct 30 21:31:20.886945 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.886730 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv" event={"ID":"c31751bb-34fe-4e1f-8bcf-2f9b457d3379","Type":"ContainerDied","Data":"79abc38be52276b7b78ea55587accc5898eba1255a834228c8988dd7d8505664"} Oct 30 21:31:20.886945 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.886826 4848 scope.go:122] "RemoveContainer" containerID="d92f44dd44cd4143c49b99d280d55b31616f254101690807bd13b10a878f58d2" Oct 30 21:31:20.925891 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.925851 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv"] Oct 30 21:31:20.937934 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.937899 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="127e9b91-a76e-46f6-9970-e5dcf8a0ba78" path="/var/lib/kubelet/pods/127e9b91-a76e-46f6-9970-e5dcf8a0ba78/volumes" Oct 30 21:31:20.938283 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:20.938268 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5506f4-openshift-default-75fcfff45c-r6tmv"] Oct 30 21:31:21.051799 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.032566 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr"] Oct 30 21:31:21.082788 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.076109 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" podUID="1f478d7b-4805-4559-93cb-33e622b3a537" containerName="discovery" containerID="cri-o://aa1c06a37cef2b28eafb1a2742ded5535b1745645655f180451bb671526a8a7c" gracePeriod=30 Oct 30 21:31:21.427281 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.427215 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 21:31:21.482525 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.482487 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f478d7b-4805-4559-93cb-33e622b3a537-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f478d7b-4805-4559-93cb-33e622b3a537-local-certs\") pod \"1f478d7b-4805-4559-93cb-33e622b3a537\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " Oct 30 21:31:21.482734 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.482593 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f478d7b-4805-4559-93cb-33e622b3a537-kube-api-access-mk46c\" (UniqueName: \"kubernetes.io/projected/1f478d7b-4805-4559-93cb-33e622b3a537-kube-api-access-mk46c\") pod \"1f478d7b-4805-4559-93cb-33e622b3a537\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " Oct 30 21:31:21.482734 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.482634 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-istio-kubeconfig\") pod \"1f478d7b-4805-4559-93cb-33e622b3a537\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " Oct 30 21:31:21.482734 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.482724 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-istio-csr-dns-cert\") pod \"1f478d7b-4805-4559-93cb-33e622b3a537\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " Oct 30 21:31:21.482963 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.482836 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f478d7b-4805-4559-93cb-33e622b3a537-istio-token\" (UniqueName: \"kubernetes.io/projected/1f478d7b-4805-4559-93cb-33e622b3a537-istio-token\") pod \"1f478d7b-4805-4559-93cb-33e622b3a537\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " Oct 30 21:31:21.482963 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.482883 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-cacerts\" (UniqueName: \"kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-cacerts\") pod \"1f478d7b-4805-4559-93cb-33e622b3a537\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " Oct 30 21:31:21.482963 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.482931 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f478d7b-4805-4559-93cb-33e622b3a537-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f478d7b-4805-4559-93cb-33e622b3a537-istio-csr-ca-configmap\") pod \"1f478d7b-4805-4559-93cb-33e622b3a537\" (UID: \"1f478d7b-4805-4559-93cb-33e622b3a537\") " Oct 30 21:31:21.483933 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.483903 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f478d7b-4805-4559-93cb-33e622b3a537-istio-csr-ca-configmap" pod "1f478d7b-4805-4559-93cb-33e622b3a537" (UID: "1f478d7b-4805-4559-93cb-33e622b3a537"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:21.486336 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.486307 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-istio-kubeconfig" pod "1f478d7b-4805-4559-93cb-33e622b3a537" (UID: "1f478d7b-4805-4559-93cb-33e622b3a537"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:21.486472 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.486447 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f478d7b-4805-4559-93cb-33e622b3a537-istio-token" pod "1f478d7b-4805-4559-93cb-33e622b3a537" (UID: "1f478d7b-4805-4559-93cb-33e622b3a537"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:21.486601 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.486574 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-istio-csr-dns-cert" pod "1f478d7b-4805-4559-93cb-33e622b3a537" (UID: "1f478d7b-4805-4559-93cb-33e622b3a537"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:21.486720 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.486696 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f478d7b-4805-4559-93cb-33e622b3a537-local-certs" pod "1f478d7b-4805-4559-93cb-33e622b3a537" (UID: "1f478d7b-4805-4559-93cb-33e622b3a537"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:21.486845 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.486747 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f478d7b-4805-4559-93cb-33e622b3a537-kube-api-access-mk46c" pod "1f478d7b-4805-4559-93cb-33e622b3a537" (UID: "1f478d7b-4805-4559-93cb-33e622b3a537"). InnerVolumeSpecName "kube-api-access-mk46c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:21.488807 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.488214 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-cacerts" pod "1f478d7b-4805-4559-93cb-33e622b3a537" (UID: "1f478d7b-4805-4559-93cb-33e622b3a537"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:21.584738 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.584703 4848 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f478d7b-4805-4559-93cb-33e622b3a537-local-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:21.584738 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.584733 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk46c\" (UniqueName: \"kubernetes.io/projected/1f478d7b-4805-4559-93cb-33e622b3a537-kube-api-access-mk46c\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:21.584738 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.584745 4848 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-istio-kubeconfig\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:21.585074 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.584753 4848 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-istio-csr-dns-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:21.585074 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.584777 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f478d7b-4805-4559-93cb-33e622b3a537-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:21.585074 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.584786 4848 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1f478d7b-4805-4559-93cb-33e622b3a537-cacerts\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:21.585074 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.584794 4848 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1f478d7b-4805-4559-93cb-33e622b3a537-istio-csr-ca-configmap\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:21.895185 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.895101 4848 generic.go:356] "Generic (PLEG): container finished" podID="1f478d7b-4805-4559-93cb-33e622b3a537" containerID="aa1c06a37cef2b28eafb1a2742ded5535b1745645655f180451bb671526a8a7c" exitCode=0 Oct 30 21:31:21.895185 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.895154 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" Oct 30 21:31:21.895185 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.895178 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" event={"ID":"1f478d7b-4805-4559-93cb-33e622b3a537","Type":"ContainerDied","Data":"aa1c06a37cef2b28eafb1a2742ded5535b1745645655f180451bb671526a8a7c"} Oct 30 21:31:21.895499 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.895225 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr" event={"ID":"1f478d7b-4805-4559-93cb-33e622b3a537","Type":"ContainerDied","Data":"fe6a988f5773e981e40a04aeac95575d1fae9ab31bef2ec2bc6beea80b9ff554"} Oct 30 21:31:21.895499 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.895247 4848 scope.go:122] "RemoveContainer" containerID="aa1c06a37cef2b28eafb1a2742ded5535b1745645655f180451bb671526a8a7c" Oct 30 21:31:21.913069 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.913039 4848 scope.go:122] "RemoveContainer" containerID="aa1c06a37cef2b28eafb1a2742ded5535b1745645655f180451bb671526a8a7c" Oct 30 21:31:21.913396 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:31:21.913372 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1c06a37cef2b28eafb1a2742ded5535b1745645655f180451bb671526a8a7c\": container with ID starting with aa1c06a37cef2b28eafb1a2742ded5535b1745645655f180451bb671526a8a7c not found: ID does not exist" containerID="aa1c06a37cef2b28eafb1a2742ded5535b1745645655f180451bb671526a8a7c" Oct 30 21:31:21.913484 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.913404 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1c06a37cef2b28eafb1a2742ded5535b1745645655f180451bb671526a8a7c"} err="failed to get container status \"aa1c06a37cef2b28eafb1a2742ded5535b1745645655f180451bb671526a8a7c\": rpc error: code = NotFound desc = could not find container \"aa1c06a37cef2b28eafb1a2742ded5535b1745645655f180451bb671526a8a7c\": container with ID starting with aa1c06a37cef2b28eafb1a2742ded5535b1745645655f180451bb671526a8a7c not found: ID does not exist" Oct 30 21:31:21.926118 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.926087 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr"] Oct 30 21:31:21.929108 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:21.929083 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qm4kr"] Oct 30 21:31:22.513518 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:31:22.513483 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:31:22.933144 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:22.933110 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f478d7b-4805-4559-93cb-33e622b3a537" path="/var/lib/kubelet/pods/1f478d7b-4805-4559-93cb-33e622b3a537/volumes" Oct 30 21:31:22.933650 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:22.933629 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c31751bb-34fe-4e1f-8bcf-2f9b457d3379" path="/var/lib/kubelet/pods/c31751bb-34fe-4e1f-8bcf-2f9b457d3379/volumes" Oct 30 21:31:23.379453 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:23.379366 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68"] Oct 30 21:31:23.379662 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:23.379615 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" podUID="248aa1ff-dd7f-455d-a0db-8881a9af59da" containerName="maas-api" containerID="cri-o://4701e452041447b409e68b7cf5f2a40594bdc93f0fa83d3fe3bfe70ab05a0261" gracePeriod=30 Oct 30 21:31:23.785820 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:23.785789 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" Oct 30 21:31:23.914098 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:23.914058 4848 generic.go:356] "Generic (PLEG): container finished" podID="248aa1ff-dd7f-455d-a0db-8881a9af59da" containerID="4701e452041447b409e68b7cf5f2a40594bdc93f0fa83d3fe3bfe70ab05a0261" exitCode=0 Oct 30 21:31:23.914282 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:23.914131 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" Oct 30 21:31:23.914282 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:23.914150 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" event={"ID":"248aa1ff-dd7f-455d-a0db-8881a9af59da","Type":"ContainerDied","Data":"4701e452041447b409e68b7cf5f2a40594bdc93f0fa83d3fe3bfe70ab05a0261"} Oct 30 21:31:23.914282 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:23.914209 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68" event={"ID":"248aa1ff-dd7f-455d-a0db-8881a9af59da","Type":"ContainerDied","Data":"53019e5f6ffb1e60bcc35c93d0c0afdfdd07142ddb8e109c488e3cf1008f576d"} Oct 30 21:31:23.914282 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:23.914238 4848 scope.go:122] "RemoveContainer" containerID="4701e452041447b409e68b7cf5f2a40594bdc93f0fa83d3fe3bfe70ab05a0261" Oct 30 21:31:23.937614 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:23.937576 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/248aa1ff-dd7f-455d-a0db-8881a9af59da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/248aa1ff-dd7f-455d-a0db-8881a9af59da-metrics-certs\") pod \"248aa1ff-dd7f-455d-a0db-8881a9af59da\" (UID: \"248aa1ff-dd7f-455d-a0db-8881a9af59da\") " Oct 30 21:31:23.937982 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:23.937943 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/248aa1ff-dd7f-455d-a0db-8881a9af59da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/248aa1ff-dd7f-455d-a0db-8881a9af59da-openshift-service-ca\") pod \"248aa1ff-dd7f-455d-a0db-8881a9af59da\" (UID: \"248aa1ff-dd7f-455d-a0db-8881a9af59da\") " Oct 30 21:31:23.938134 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:23.938109 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/248aa1ff-dd7f-455d-a0db-8881a9af59da-kube-api-access-tkl7l\" (UniqueName: \"kubernetes.io/projected/248aa1ff-dd7f-455d-a0db-8881a9af59da-kube-api-access-tkl7l\") pod \"248aa1ff-dd7f-455d-a0db-8881a9af59da\" (UID: \"248aa1ff-dd7f-455d-a0db-8881a9af59da\") " Oct 30 21:31:23.938449 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:23.938427 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/248aa1ff-dd7f-455d-a0db-8881a9af59da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/248aa1ff-dd7f-455d-a0db-8881a9af59da-maas-api-tls\") pod \"248aa1ff-dd7f-455d-a0db-8881a9af59da\" (UID: \"248aa1ff-dd7f-455d-a0db-8881a9af59da\") " Oct 30 21:31:23.948197 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:23.940278 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248aa1ff-dd7f-455d-a0db-8881a9af59da-openshift-service-ca" pod "248aa1ff-dd7f-455d-a0db-8881a9af59da" (UID: "248aa1ff-dd7f-455d-a0db-8881a9af59da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:23.950372 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:23.950308 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/248aa1ff-dd7f-455d-a0db-8881a9af59da-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:23.952010 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:23.951972 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248aa1ff-dd7f-455d-a0db-8881a9af59da-kube-api-access-tkl7l" pod "248aa1ff-dd7f-455d-a0db-8881a9af59da" (UID: "248aa1ff-dd7f-455d-a0db-8881a9af59da"). InnerVolumeSpecName "kube-api-access-tkl7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:23.952340 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:23.952289 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/248aa1ff-dd7f-455d-a0db-8881a9af59da-maas-api-tls" pod "248aa1ff-dd7f-455d-a0db-8881a9af59da" (UID: "248aa1ff-dd7f-455d-a0db-8881a9af59da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:23.952593 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:23.952567 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/248aa1ff-dd7f-455d-a0db-8881a9af59da-metrics-certs" pod "248aa1ff-dd7f-455d-a0db-8881a9af59da" (UID: "248aa1ff-dd7f-455d-a0db-8881a9af59da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:23.961739 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:23.961714 4848 scope.go:122] "RemoveContainer" containerID="4701e452041447b409e68b7cf5f2a40594bdc93f0fa83d3fe3bfe70ab05a0261" Oct 30 21:31:23.962087 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:31:23.962061 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4701e452041447b409e68b7cf5f2a40594bdc93f0fa83d3fe3bfe70ab05a0261\": container with ID starting with 4701e452041447b409e68b7cf5f2a40594bdc93f0fa83d3fe3bfe70ab05a0261 not found: ID does not exist" containerID="4701e452041447b409e68b7cf5f2a40594bdc93f0fa83d3fe3bfe70ab05a0261" Oct 30 21:31:23.962183 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:23.962100 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4701e452041447b409e68b7cf5f2a40594bdc93f0fa83d3fe3bfe70ab05a0261"} err="failed to get container status \"4701e452041447b409e68b7cf5f2a40594bdc93f0fa83d3fe3bfe70ab05a0261\": rpc error: code = NotFound desc = could not find container \"4701e452041447b409e68b7cf5f2a40594bdc93f0fa83d3fe3bfe70ab05a0261\": container with ID starting with 4701e452041447b409e68b7cf5f2a40594bdc93f0fa83d3fe3bfe70ab05a0261 not found: ID does not exist" Oct 30 21:31:24.052348 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:24.052237 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/248aa1ff-dd7f-455d-a0db-8881a9af59da-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:24.052348 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:24.052284 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkl7l\" (UniqueName: \"kubernetes.io/projected/248aa1ff-dd7f-455d-a0db-8881a9af59da-kube-api-access-tkl7l\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:24.052348 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:24.052300 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/248aa1ff-dd7f-455d-a0db-8881a9af59da-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:24.241098 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:24.241066 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68"] Oct 30 21:31:24.247752 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:24.247726 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e1e0f5-85957f9fbc-s9t68"] Oct 30 21:31:24.482833 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:24.482798 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj"] Oct 30 21:31:24.932336 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:24.932299 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="248aa1ff-dd7f-455d-a0db-8881a9af59da" path="/var/lib/kubelet/pods/248aa1ff-dd7f-455d-a0db-8881a9af59da/volumes" Oct 30 21:31:28.766362 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:28.766322 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg"] Oct 30 21:31:28.766817 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:28.766584 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" podUID="9d3d52ea-ca0a-456e-bf85-896e16fedc84" containerName="maas-api" containerID="cri-o://26716bf6484e87cb4548f364a848166c75e1c11a572662f3b122b2f3bf469280" gracePeriod=30 Oct 30 21:31:28.957405 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:28.949661 4848 generic.go:356] "Generic (PLEG): container finished" podID="9d3d52ea-ca0a-456e-bf85-896e16fedc84" containerID="26716bf6484e87cb4548f364a848166c75e1c11a572662f3b122b2f3bf469280" exitCode=0 Oct 30 21:31:28.957405 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:28.949818 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" event={"ID":"9d3d52ea-ca0a-456e-bf85-896e16fedc84","Type":"ContainerDied","Data":"26716bf6484e87cb4548f364a848166c75e1c11a572662f3b122b2f3bf469280"} Oct 30 21:31:29.161643 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:29.161615 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" Oct 30 21:31:29.240275 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:29.240241 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d3d52ea-ca0a-456e-bf85-896e16fedc84-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d3d52ea-ca0a-456e-bf85-896e16fedc84-openshift-service-ca\") pod \"9d3d52ea-ca0a-456e-bf85-896e16fedc84\" (UID: \"9d3d52ea-ca0a-456e-bf85-896e16fedc84\") " Oct 30 21:31:29.240460 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:29.240292 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3d52ea-ca0a-456e-bf85-896e16fedc84-kube-api-access-zj8fh\" (UniqueName: \"kubernetes.io/projected/9d3d52ea-ca0a-456e-bf85-896e16fedc84-kube-api-access-zj8fh\") pod \"9d3d52ea-ca0a-456e-bf85-896e16fedc84\" (UID: \"9d3d52ea-ca0a-456e-bf85-896e16fedc84\") " Oct 30 21:31:29.240460 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:29.240371 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d3d52ea-ca0a-456e-bf85-896e16fedc84-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d3d52ea-ca0a-456e-bf85-896e16fedc84-maas-api-tls\") pod \"9d3d52ea-ca0a-456e-bf85-896e16fedc84\" (UID: \"9d3d52ea-ca0a-456e-bf85-896e16fedc84\") " Oct 30 21:31:29.240460 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:29.240414 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d3d52ea-ca0a-456e-bf85-896e16fedc84-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d3d52ea-ca0a-456e-bf85-896e16fedc84-metrics-certs\") pod \"9d3d52ea-ca0a-456e-bf85-896e16fedc84\" (UID: \"9d3d52ea-ca0a-456e-bf85-896e16fedc84\") " Oct 30 21:31:29.240637 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:29.240615 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d3d52ea-ca0a-456e-bf85-896e16fedc84-openshift-service-ca" pod "9d3d52ea-ca0a-456e-bf85-896e16fedc84" (UID: "9d3d52ea-ca0a-456e-bf85-896e16fedc84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:29.240929 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:29.240910 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d3d52ea-ca0a-456e-bf85-896e16fedc84-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:29.242465 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:29.242436 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d3d52ea-ca0a-456e-bf85-896e16fedc84-metrics-certs" pod "9d3d52ea-ca0a-456e-bf85-896e16fedc84" (UID: "9d3d52ea-ca0a-456e-bf85-896e16fedc84"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:29.242572 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:29.242499 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3d52ea-ca0a-456e-bf85-896e16fedc84-kube-api-access-zj8fh" pod "9d3d52ea-ca0a-456e-bf85-896e16fedc84" (UID: "9d3d52ea-ca0a-456e-bf85-896e16fedc84"). InnerVolumeSpecName "kube-api-access-zj8fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:29.242572 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:29.242509 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d3d52ea-ca0a-456e-bf85-896e16fedc84-maas-api-tls" pod "9d3d52ea-ca0a-456e-bf85-896e16fedc84" (UID: "9d3d52ea-ca0a-456e-bf85-896e16fedc84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:29.351843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:29.343476 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d3d52ea-ca0a-456e-bf85-896e16fedc84-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:29.351843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:29.343534 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d3d52ea-ca0a-456e-bf85-896e16fedc84-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:29.351843 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:29.343551 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj8fh\" (UniqueName: \"kubernetes.io/projected/9d3d52ea-ca0a-456e-bf85-896e16fedc84-kube-api-access-zj8fh\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:29.484110 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:29.484066 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj" podUID="e2400054-2c0b-4ade-874c-e752a9067b54" containerName="payload-pre-processing" containerID="cri-o://b5629ca2fe118e92a693396af5c0f61dbfadccf2229a846f82ed9a8a109eae52" gracePeriod=25 Oct 30 21:31:29.958687 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:29.958660 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" Oct 30 21:31:29.958687 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:29.958668 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg" event={"ID":"9d3d52ea-ca0a-456e-bf85-896e16fedc84","Type":"ContainerDied","Data":"46aee9ba2944af28e80d719179f911ccdcc922d8a68b5bd6d9adcc84df2d5146"} Oct 30 21:31:29.959260 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:29.958745 4848 scope.go:122] "RemoveContainer" containerID="26716bf6484e87cb4548f364a848166c75e1c11a572662f3b122b2f3bf469280" Oct 30 21:31:29.982718 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:29.982659 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg"] Oct 30 21:31:29.986878 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:29.986853 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-e07442-b96bddd9-sfljg"] Oct 30 21:31:30.933880 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:30.933846 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d3d52ea-ca0a-456e-bf85-896e16fedc84" path="/var/lib/kubelet/pods/9d3d52ea-ca0a-456e-bf85-896e16fedc84/volumes" Oct 30 21:31:38.050378 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:38.050292 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565"] Oct 30 21:31:38.050979 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:38.050676 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" podUID="f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" containerName="istio-proxy" containerID="cri-o://721652c39a3bca0f03817f8f78389b303cb87368a19c36037f47fec2562ccfc4" gracePeriod=30 Oct 30 21:31:39.200040 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:39.200005 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj" podUID="e2400054-2c0b-4ade-874c-e752a9067b54" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:31:43.362176 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.362148 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:31:43.537403 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.537367 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-kube-api-access-bfjwn\" (UniqueName: \"kubernetes.io/projected/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-kube-api-access-bfjwn\") pod \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " Oct 30 21:31:43.537610 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.537410 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-podinfo\") pod \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " Oct 30 21:31:43.537610 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.537470 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-envoy\") pod \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " Oct 30 21:31:43.537610 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.537494 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-token\" (UniqueName: \"kubernetes.io/projected/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-token\") pod \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " Oct 30 21:31:43.537610 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.537519 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-workload-socket\") pod \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " Oct 30 21:31:43.537610 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.537567 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-data\") pod \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " Oct 30 21:31:43.537610 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.537587 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istiod-ca-cert\") pod \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " Oct 30 21:31:43.538013 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.537692 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-credential-socket\") pod \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " Oct 30 21:31:43.538013 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.537802 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-workload-certs\") pod \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\" (UID: \"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8\") " Oct 30 21:31:43.538013 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.537991 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-data" pod "f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" (UID: "f6f0c012-836a-4de1-b9a8-2acd35ec9ce8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:43.538224 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.538200 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-credential-socket" pod "f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" (UID: "f6f0c012-836a-4de1-b9a8-2acd35ec9ce8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:43.538224 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.538119 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istiod-ca-cert" pod "f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" (UID: "f6f0c012-836a-4de1-b9a8-2acd35ec9ce8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:43.538423 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.538385 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-workload-certs" pod "f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" (UID: "f6f0c012-836a-4de1-b9a8-2acd35ec9ce8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:43.538508 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.538435 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-workload-socket" pod "f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" (UID: "f6f0c012-836a-4de1-b9a8-2acd35ec9ce8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:43.538508 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.538505 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:43.538629 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.538520 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:43.538629 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.538535 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:43.539855 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.539833 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-token" pod "f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" (UID: "f6f0c012-836a-4de1-b9a8-2acd35ec9ce8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:43.540030 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.540011 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-podinfo" pod "f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" (UID: "f6f0c012-836a-4de1-b9a8-2acd35ec9ce8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:31:43.540124 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.540050 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-envoy" pod "f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" (UID: "f6f0c012-836a-4de1-b9a8-2acd35ec9ce8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:43.540124 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.540081 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-kube-api-access-bfjwn" pod "f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" (UID: "f6f0c012-836a-4de1-b9a8-2acd35ec9ce8"). InnerVolumeSpecName "kube-api-access-bfjwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:43.640234 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.640198 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:43.640234 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.640230 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfjwn\" (UniqueName: \"kubernetes.io/projected/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-kube-api-access-bfjwn\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:43.640234 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.640241 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:43.640488 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.640249 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:43.640488 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.640257 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:43.640488 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:43.640266 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:44.115602 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:44.115570 4848 generic.go:356] "Generic (PLEG): container finished" podID="f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" containerID="721652c39a3bca0f03817f8f78389b303cb87368a19c36037f47fec2562ccfc4" exitCode=0 Oct 30 21:31:44.115750 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:44.115612 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" event={"ID":"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8","Type":"ContainerDied","Data":"721652c39a3bca0f03817f8f78389b303cb87368a19c36037f47fec2562ccfc4"} Oct 30 21:31:44.115750 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:44.115625 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" Oct 30 21:31:44.115750 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:44.115655 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565" event={"ID":"f6f0c012-836a-4de1-b9a8-2acd35ec9ce8","Type":"ContainerDied","Data":"3e7b31ee45a1e4208fd9e29616bea23fb1afdb3e9e33ec30f44c664d2015bf90"} Oct 30 21:31:44.115750 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:44.115676 4848 scope.go:122] "RemoveContainer" containerID="721652c39a3bca0f03817f8f78389b303cb87368a19c36037f47fec2562ccfc4" Oct 30 21:31:44.127416 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:44.127396 4848 scope.go:122] "RemoveContainer" containerID="a913616692ba833bd5b763de36ee7a8958aa5799d71eac70485a1d403c55999e" Oct 30 21:31:44.140176 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:44.140159 4848 scope.go:122] "RemoveContainer" containerID="721652c39a3bca0f03817f8f78389b303cb87368a19c36037f47fec2562ccfc4" Oct 30 21:31:44.140512 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:31:44.140492 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"721652c39a3bca0f03817f8f78389b303cb87368a19c36037f47fec2562ccfc4\": container with ID starting with 721652c39a3bca0f03817f8f78389b303cb87368a19c36037f47fec2562ccfc4 not found: ID does not exist" containerID="721652c39a3bca0f03817f8f78389b303cb87368a19c36037f47fec2562ccfc4" Oct 30 21:31:44.140575 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:44.140521 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"721652c39a3bca0f03817f8f78389b303cb87368a19c36037f47fec2562ccfc4"} err="failed to get container status \"721652c39a3bca0f03817f8f78389b303cb87368a19c36037f47fec2562ccfc4\": rpc error: code = NotFound desc = could not find container \"721652c39a3bca0f03817f8f78389b303cb87368a19c36037f47fec2562ccfc4\": container with ID starting with 721652c39a3bca0f03817f8f78389b303cb87368a19c36037f47fec2562ccfc4 not found: ID does not exist" Oct 30 21:31:44.140575 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:44.140539 4848 scope.go:122] "RemoveContainer" containerID="a913616692ba833bd5b763de36ee7a8958aa5799d71eac70485a1d403c55999e" Oct 30 21:31:44.140848 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:31:44.140829 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a913616692ba833bd5b763de36ee7a8958aa5799d71eac70485a1d403c55999e\": container with ID starting with a913616692ba833bd5b763de36ee7a8958aa5799d71eac70485a1d403c55999e not found: ID does not exist" containerID="a913616692ba833bd5b763de36ee7a8958aa5799d71eac70485a1d403c55999e" Oct 30 21:31:44.140902 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:44.140855 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a913616692ba833bd5b763de36ee7a8958aa5799d71eac70485a1d403c55999e"} err="failed to get container status \"a913616692ba833bd5b763de36ee7a8958aa5799d71eac70485a1d403c55999e\": rpc error: code = NotFound desc = could not find container \"a913616692ba833bd5b763de36ee7a8958aa5799d71eac70485a1d403c55999e\": container with ID starting with a913616692ba833bd5b763de36ee7a8958aa5799d71eac70485a1d403c55999e not found: ID does not exist" Oct 30 21:31:44.141478 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:44.141460 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565"] Oct 30 21:31:44.145298 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:44.145276 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e07442-openshift-default-56d77688d5-wr565"] Oct 30 21:31:44.936278 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:44.936244 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" path="/var/lib/kubelet/pods/f6f0c012-836a-4de1-b9a8-2acd35ec9ce8/volumes" Oct 30 21:31:47.464008 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:47.463977 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg"] Oct 30 21:31:47.464544 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:47.464290 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" podUID="3974a64c-bb4d-436e-bcfe-da5277f71f8c" containerName="maas-api" containerID="cri-o://4bf13d21f81a2bfc1226d7f0ef0d022fb5ddd4233e63a418035e6e15769be911" gracePeriod=30 Oct 30 21:31:47.892892 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:47.892858 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" Oct 30 21:31:48.010456 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.010421 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3974a64c-bb4d-436e-bcfe-da5277f71f8c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3974a64c-bb4d-436e-bcfe-da5277f71f8c-openshift-service-ca\") pod \"3974a64c-bb4d-436e-bcfe-da5277f71f8c\" (UID: \"3974a64c-bb4d-436e-bcfe-da5277f71f8c\") " Oct 30 21:31:48.010652 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.010525 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3974a64c-bb4d-436e-bcfe-da5277f71f8c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3974a64c-bb4d-436e-bcfe-da5277f71f8c-maas-api-tls\") pod \"3974a64c-bb4d-436e-bcfe-da5277f71f8c\" (UID: \"3974a64c-bb4d-436e-bcfe-da5277f71f8c\") " Oct 30 21:31:48.010652 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.010557 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3974a64c-bb4d-436e-bcfe-da5277f71f8c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3974a64c-bb4d-436e-bcfe-da5277f71f8c-metrics-certs\") pod \"3974a64c-bb4d-436e-bcfe-da5277f71f8c\" (UID: \"3974a64c-bb4d-436e-bcfe-da5277f71f8c\") " Oct 30 21:31:48.010830 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.010653 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3974a64c-bb4d-436e-bcfe-da5277f71f8c-kube-api-access-26q7r\" (UniqueName: \"kubernetes.io/projected/3974a64c-bb4d-436e-bcfe-da5277f71f8c-kube-api-access-26q7r\") pod \"3974a64c-bb4d-436e-bcfe-da5277f71f8c\" (UID: \"3974a64c-bb4d-436e-bcfe-da5277f71f8c\") " Oct 30 21:31:48.010902 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.010829 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3974a64c-bb4d-436e-bcfe-da5277f71f8c-openshift-service-ca" pod "3974a64c-bb4d-436e-bcfe-da5277f71f8c" (UID: "3974a64c-bb4d-436e-bcfe-da5277f71f8c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:48.011243 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.011218 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3974a64c-bb4d-436e-bcfe-da5277f71f8c-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:48.012891 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.012869 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3974a64c-bb4d-436e-bcfe-da5277f71f8c-maas-api-tls" pod "3974a64c-bb4d-436e-bcfe-da5277f71f8c" (UID: "3974a64c-bb4d-436e-bcfe-da5277f71f8c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:48.012996 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.012897 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3974a64c-bb4d-436e-bcfe-da5277f71f8c-metrics-certs" pod "3974a64c-bb4d-436e-bcfe-da5277f71f8c" (UID: "3974a64c-bb4d-436e-bcfe-da5277f71f8c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:48.013099 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.013082 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3974a64c-bb4d-436e-bcfe-da5277f71f8c-kube-api-access-26q7r" pod "3974a64c-bb4d-436e-bcfe-da5277f71f8c" (UID: "3974a64c-bb4d-436e-bcfe-da5277f71f8c"). InnerVolumeSpecName "kube-api-access-26q7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:48.112792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.112689 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3974a64c-bb4d-436e-bcfe-da5277f71f8c-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:48.112792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.112718 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3974a64c-bb4d-436e-bcfe-da5277f71f8c-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:48.112792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.112736 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26q7r\" (UniqueName: \"kubernetes.io/projected/3974a64c-bb4d-436e-bcfe-da5277f71f8c-kube-api-access-26q7r\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:48.154598 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.154564 4848 generic.go:356] "Generic (PLEG): container finished" podID="3974a64c-bb4d-436e-bcfe-da5277f71f8c" containerID="4bf13d21f81a2bfc1226d7f0ef0d022fb5ddd4233e63a418035e6e15769be911" exitCode=0 Oct 30 21:31:48.154824 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.154607 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" Oct 30 21:31:48.154824 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.154649 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" event={"ID":"3974a64c-bb4d-436e-bcfe-da5277f71f8c","Type":"ContainerDied","Data":"4bf13d21f81a2bfc1226d7f0ef0d022fb5ddd4233e63a418035e6e15769be911"} Oct 30 21:31:48.154824 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.154710 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg" event={"ID":"3974a64c-bb4d-436e-bcfe-da5277f71f8c","Type":"ContainerDied","Data":"e8f3ed87dbca4ce26f8783607272ef8a9c7f22da2e43d70ff0cfe8da3458bfc6"} Oct 30 21:31:48.154824 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.154739 4848 scope.go:122] "RemoveContainer" containerID="4bf13d21f81a2bfc1226d7f0ef0d022fb5ddd4233e63a418035e6e15769be911" Oct 30 21:31:48.165443 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.165423 4848 scope.go:122] "RemoveContainer" containerID="4bf13d21f81a2bfc1226d7f0ef0d022fb5ddd4233e63a418035e6e15769be911" Oct 30 21:31:48.165733 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:31:48.165712 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bf13d21f81a2bfc1226d7f0ef0d022fb5ddd4233e63a418035e6e15769be911\": container with ID starting with 4bf13d21f81a2bfc1226d7f0ef0d022fb5ddd4233e63a418035e6e15769be911 not found: ID does not exist" containerID="4bf13d21f81a2bfc1226d7f0ef0d022fb5ddd4233e63a418035e6e15769be911" Oct 30 21:31:48.165837 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.165742 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf13d21f81a2bfc1226d7f0ef0d022fb5ddd4233e63a418035e6e15769be911"} err="failed to get container status \"4bf13d21f81a2bfc1226d7f0ef0d022fb5ddd4233e63a418035e6e15769be911\": rpc error: code = NotFound desc = could not find container \"4bf13d21f81a2bfc1226d7f0ef0d022fb5ddd4233e63a418035e6e15769be911\": container with ID starting with 4bf13d21f81a2bfc1226d7f0ef0d022fb5ddd4233e63a418035e6e15769be911 not found: ID does not exist" Oct 30 21:31:48.191041 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.191006 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg"] Oct 30 21:31:48.195660 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.195633 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-3a65ec-5f68bb9855-t8kdg"] Oct 30 21:31:48.932408 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:48.932373 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3974a64c-bb4d-436e-bcfe-da5277f71f8c" path="/var/lib/kubelet/pods/3974a64c-bb4d-436e-bcfe-da5277f71f8c/volumes" Oct 30 21:31:49.198323 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:49.198241 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj" podUID="e2400054-2c0b-4ade-874c-e752a9067b54" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:31:49.747911 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:49.747852 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj" Oct 30 21:31:49.835898 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:49.835858 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2400054-2c0b-4ade-874c-e752a9067b54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2400054-2c0b-4ade-874c-e752a9067b54-plugins-config-volume\") pod \"e2400054-2c0b-4ade-874c-e752a9067b54\" (UID: \"e2400054-2c0b-4ade-874c-e752a9067b54\") " Oct 30 21:31:49.837056 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:49.837016 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2400054-2c0b-4ade-874c-e752a9067b54-kube-api-access-2wm57\" (UniqueName: \"kubernetes.io/projected/e2400054-2c0b-4ade-874c-e752a9067b54-kube-api-access-2wm57\") pod \"e2400054-2c0b-4ade-874c-e752a9067b54\" (UID: \"e2400054-2c0b-4ade-874c-e752a9067b54\") " Oct 30 21:31:49.838591 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:49.838568 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2400054-2c0b-4ade-874c-e752a9067b54-plugins-config-volume" pod "e2400054-2c0b-4ade-874c-e752a9067b54" (UID: "e2400054-2c0b-4ade-874c-e752a9067b54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:49.840320 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:49.840282 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2400054-2c0b-4ade-874c-e752a9067b54-kube-api-access-2wm57" pod "e2400054-2c0b-4ade-874c-e752a9067b54" (UID: "e2400054-2c0b-4ade-874c-e752a9067b54"). InnerVolumeSpecName "kube-api-access-2wm57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:49.939332 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:49.939299 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wm57\" (UniqueName: \"kubernetes.io/projected/e2400054-2c0b-4ade-874c-e752a9067b54-kube-api-access-2wm57\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:49.939332 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:49.939327 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2400054-2c0b-4ade-874c-e752a9067b54-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:50.185692 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:50.185656 4848 generic.go:356] "Generic (PLEG): container finished" podID="e2400054-2c0b-4ade-874c-e752a9067b54" containerID="b5629ca2fe118e92a693396af5c0f61dbfadccf2229a846f82ed9a8a109eae52" exitCode=0 Oct 30 21:31:50.185925 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:50.185707 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj" Oct 30 21:31:50.185925 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:50.185714 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj" event={"ID":"e2400054-2c0b-4ade-874c-e752a9067b54","Type":"ContainerDied","Data":"b5629ca2fe118e92a693396af5c0f61dbfadccf2229a846f82ed9a8a109eae52"} Oct 30 21:31:50.185925 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:50.185785 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj" event={"ID":"e2400054-2c0b-4ade-874c-e752a9067b54","Type":"ContainerDied","Data":"819e141b55a94e5a01bca602ef7a2fe22a9eab518b498e27b86367618e3925cd"} Oct 30 21:31:50.185925 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:50.185816 4848 scope.go:122] "RemoveContainer" containerID="b5629ca2fe118e92a693396af5c0f61dbfadccf2229a846f82ed9a8a109eae52" Oct 30 21:31:50.196891 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:50.196858 4848 scope.go:122] "RemoveContainer" containerID="b5629ca2fe118e92a693396af5c0f61dbfadccf2229a846f82ed9a8a109eae52" Oct 30 21:31:50.197808 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:31:50.197223 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5629ca2fe118e92a693396af5c0f61dbfadccf2229a846f82ed9a8a109eae52\": container with ID starting with b5629ca2fe118e92a693396af5c0f61dbfadccf2229a846f82ed9a8a109eae52 not found: ID does not exist" containerID="b5629ca2fe118e92a693396af5c0f61dbfadccf2229a846f82ed9a8a109eae52" Oct 30 21:31:50.197808 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:50.197266 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5629ca2fe118e92a693396af5c0f61dbfadccf2229a846f82ed9a8a109eae52"} err="failed to get container status \"b5629ca2fe118e92a693396af5c0f61dbfadccf2229a846f82ed9a8a109eae52\": rpc error: code = NotFound desc = could not find container \"b5629ca2fe118e92a693396af5c0f61dbfadccf2229a846f82ed9a8a109eae52\": container with ID starting with b5629ca2fe118e92a693396af5c0f61dbfadccf2229a846f82ed9a8a109eae52 not found: ID does not exist" Oct 30 21:31:50.213614 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:50.213546 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj"] Oct 30 21:31:50.216873 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:50.216835 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e07442-55fb7dd479-f5bkj"] Oct 30 21:31:50.933258 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:50.933226 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2400054-2c0b-4ade-874c-e752a9067b54" path="/var/lib/kubelet/pods/e2400054-2c0b-4ade-874c-e752a9067b54/volumes" Oct 30 21:31:52.524404 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:31:52.524363 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:31:56.574348 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:56.574306 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p"] Oct 30 21:31:56.574742 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:56.574594 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" podUID="3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb" containerName="maas-api" containerID="cri-o://4a2b6a9800e8db0fff01b274ddad8dd8ac8015eb89d295fc0338bf1f67f635f0" gracePeriod=30 Oct 30 21:31:56.945086 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:56.945052 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" Oct 30 21:31:56.968659 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:56.968629 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-metrics-certs\") pod \"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb\" (UID: \"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb\") " Oct 30 21:31:56.968891 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:56.968683 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-openshift-service-ca\") pod \"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb\" (UID: \"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb\") " Oct 30 21:31:56.968891 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:56.968721 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-kube-api-access-td4h4\" (UniqueName: \"kubernetes.io/projected/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-kube-api-access-td4h4\") pod \"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb\" (UID: \"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb\") " Oct 30 21:31:56.968891 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:56.968745 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-maas-api-tls\") pod \"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb\" (UID: \"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb\") " Oct 30 21:31:56.969178 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:56.969152 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-openshift-service-ca" pod "3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb" (UID: "3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:56.969988 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:56.969960 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:56.970971 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:56.970938 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-metrics-certs" pod "3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb" (UID: "3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:56.971076 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:56.970989 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-kube-api-access-td4h4" pod "3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb" (UID: "3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb"). InnerVolumeSpecName "kube-api-access-td4h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:56.971172 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:56.971147 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-maas-api-tls" pod "3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb" (UID: "3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:57.071037 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:57.071003 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td4h4\" (UniqueName: \"kubernetes.io/projected/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-kube-api-access-td4h4\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:57.071037 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:57.071035 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:57.071037 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:57.071044 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:57.232304 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:57.232265 4848 generic.go:356] "Generic (PLEG): container finished" podID="3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb" containerID="4a2b6a9800e8db0fff01b274ddad8dd8ac8015eb89d295fc0338bf1f67f635f0" exitCode=0 Oct 30 21:31:57.232509 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:57.232319 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" Oct 30 21:31:57.232509 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:57.232362 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" event={"ID":"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb","Type":"ContainerDied","Data":"4a2b6a9800e8db0fff01b274ddad8dd8ac8015eb89d295fc0338bf1f67f635f0"} Oct 30 21:31:57.232509 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:57.232432 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p" event={"ID":"3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb","Type":"ContainerDied","Data":"7af3b60ec852d29f64fa0ee67a77450735f48984ced9c6eedb4d25e4d2b5c8f0"} Oct 30 21:31:57.232509 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:57.232469 4848 scope.go:122] "RemoveContainer" containerID="4a2b6a9800e8db0fff01b274ddad8dd8ac8015eb89d295fc0338bf1f67f635f0" Oct 30 21:31:57.243618 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:57.243598 4848 scope.go:122] "RemoveContainer" containerID="4a2b6a9800e8db0fff01b274ddad8dd8ac8015eb89d295fc0338bf1f67f635f0" Oct 30 21:31:57.243929 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:31:57.243909 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a2b6a9800e8db0fff01b274ddad8dd8ac8015eb89d295fc0338bf1f67f635f0\": container with ID starting with 4a2b6a9800e8db0fff01b274ddad8dd8ac8015eb89d295fc0338bf1f67f635f0 not found: ID does not exist" containerID="4a2b6a9800e8db0fff01b274ddad8dd8ac8015eb89d295fc0338bf1f67f635f0" Oct 30 21:31:57.243987 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:57.243938 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a2b6a9800e8db0fff01b274ddad8dd8ac8015eb89d295fc0338bf1f67f635f0"} err="failed to get container status \"4a2b6a9800e8db0fff01b274ddad8dd8ac8015eb89d295fc0338bf1f67f635f0\": rpc error: code = NotFound desc = could not find container \"4a2b6a9800e8db0fff01b274ddad8dd8ac8015eb89d295fc0338bf1f67f635f0\": container with ID starting with 4a2b6a9800e8db0fff01b274ddad8dd8ac8015eb89d295fc0338bf1f67f635f0 not found: ID does not exist" Oct 30 21:31:57.257327 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:57.257299 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p"] Oct 30 21:31:57.261715 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:57.261689 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8fa15b-568bff5fb6-95z6p"] Oct 30 21:31:58.713529 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:58.713499 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl"] Oct 30 21:31:58.713991 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:58.713708 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" podUID="81767769-741f-4780-93d2-32406b2a6ba6" containerName="maas-api" containerID="cri-o://2a1b556246198b87bb5b6648178f296219ca8c45780b3669a51cc2d04d22628c" gracePeriod=30 Oct 30 21:31:58.936172 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:58.936136 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb" path="/var/lib/kubelet/pods/3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb/volumes" Oct 30 21:31:59.071265 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.071206 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" Oct 30 21:31:59.115976 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.115873 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81767769-741f-4780-93d2-32406b2a6ba6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81767769-741f-4780-93d2-32406b2a6ba6-maas-api-tls\") pod \"81767769-741f-4780-93d2-32406b2a6ba6\" (UID: \"81767769-741f-4780-93d2-32406b2a6ba6\") " Oct 30 21:31:59.116153 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.116059 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81767769-741f-4780-93d2-32406b2a6ba6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81767769-741f-4780-93d2-32406b2a6ba6-openshift-service-ca\") pod \"81767769-741f-4780-93d2-32406b2a6ba6\" (UID: \"81767769-741f-4780-93d2-32406b2a6ba6\") " Oct 30 21:31:59.116153 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.116106 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81767769-741f-4780-93d2-32406b2a6ba6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/81767769-741f-4780-93d2-32406b2a6ba6-metrics-certs\") pod \"81767769-741f-4780-93d2-32406b2a6ba6\" (UID: \"81767769-741f-4780-93d2-32406b2a6ba6\") " Oct 30 21:31:59.116315 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.116231 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81767769-741f-4780-93d2-32406b2a6ba6-kube-api-access-zxxtj\" (UniqueName: \"kubernetes.io/projected/81767769-741f-4780-93d2-32406b2a6ba6-kube-api-access-zxxtj\") pod \"81767769-741f-4780-93d2-32406b2a6ba6\" (UID: \"81767769-741f-4780-93d2-32406b2a6ba6\") " Oct 30 21:31:59.116848 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.116677 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81767769-741f-4780-93d2-32406b2a6ba6-openshift-service-ca" pod "81767769-741f-4780-93d2-32406b2a6ba6" (UID: "81767769-741f-4780-93d2-32406b2a6ba6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:59.130668 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.130628 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81767769-741f-4780-93d2-32406b2a6ba6-maas-api-tls" pod "81767769-741f-4780-93d2-32406b2a6ba6" (UID: "81767769-741f-4780-93d2-32406b2a6ba6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:59.130864 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.130835 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81767769-741f-4780-93d2-32406b2a6ba6-metrics-certs" pod "81767769-741f-4780-93d2-32406b2a6ba6" (UID: "81767769-741f-4780-93d2-32406b2a6ba6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:31:59.132912 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.132866 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81767769-741f-4780-93d2-32406b2a6ba6-kube-api-access-zxxtj" pod "81767769-741f-4780-93d2-32406b2a6ba6" (UID: "81767769-741f-4780-93d2-32406b2a6ba6"). InnerVolumeSpecName "kube-api-access-zxxtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:59.217713 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.217678 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81767769-741f-4780-93d2-32406b2a6ba6-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:59.217713 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.217711 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxxtj\" (UniqueName: \"kubernetes.io/projected/81767769-741f-4780-93d2-32406b2a6ba6-kube-api-access-zxxtj\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:59.217713 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.217723 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81767769-741f-4780-93d2-32406b2a6ba6-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:59.218052 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.217733 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81767769-741f-4780-93d2-32406b2a6ba6-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:31:59.255405 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.255370 4848 generic.go:356] "Generic (PLEG): container finished" podID="81767769-741f-4780-93d2-32406b2a6ba6" containerID="2a1b556246198b87bb5b6648178f296219ca8c45780b3669a51cc2d04d22628c" exitCode=0 Oct 30 21:31:59.255630 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.255415 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" event={"ID":"81767769-741f-4780-93d2-32406b2a6ba6","Type":"ContainerDied","Data":"2a1b556246198b87bb5b6648178f296219ca8c45780b3669a51cc2d04d22628c"} Oct 30 21:31:59.255630 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.255429 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" Oct 30 21:31:59.255630 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.255456 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl" event={"ID":"81767769-741f-4780-93d2-32406b2a6ba6","Type":"ContainerDied","Data":"2de7698b30eef08521ae51ee9dd71e2ca1dff90e028f63121b55621b96551c2d"} Oct 30 21:31:59.255630 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.255480 4848 scope.go:122] "RemoveContainer" containerID="2a1b556246198b87bb5b6648178f296219ca8c45780b3669a51cc2d04d22628c" Oct 30 21:31:59.265945 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.265923 4848 scope.go:122] "RemoveContainer" containerID="2a1b556246198b87bb5b6648178f296219ca8c45780b3669a51cc2d04d22628c" Oct 30 21:31:59.266298 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:31:59.266273 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a1b556246198b87bb5b6648178f296219ca8c45780b3669a51cc2d04d22628c\": container with ID starting with 2a1b556246198b87bb5b6648178f296219ca8c45780b3669a51cc2d04d22628c not found: ID does not exist" containerID="2a1b556246198b87bb5b6648178f296219ca8c45780b3669a51cc2d04d22628c" Oct 30 21:31:59.266369 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.266309 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a1b556246198b87bb5b6648178f296219ca8c45780b3669a51cc2d04d22628c"} err="failed to get container status \"2a1b556246198b87bb5b6648178f296219ca8c45780b3669a51cc2d04d22628c\": rpc error: code = NotFound desc = could not find container \"2a1b556246198b87bb5b6648178f296219ca8c45780b3669a51cc2d04d22628c\": container with ID starting with 2a1b556246198b87bb5b6648178f296219ca8c45780b3669a51cc2d04d22628c not found: ID does not exist" Oct 30 21:31:59.279638 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.279608 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl"] Oct 30 21:31:59.284586 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:31:59.284518 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f147de-7654dcb98b-6fjfl"] Oct 30 21:32:00.586147 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:00.586109 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7"] Oct 30 21:32:00.948789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:00.937617 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81767769-741f-4780-93d2-32406b2a6ba6" path="/var/lib/kubelet/pods/81767769-741f-4780-93d2-32406b2a6ba6/volumes" Oct 30 21:32:05.587172 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:05.587072 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7" podUID="fa84afb2-c58a-4ac6-be54-bdfef411fd2b" containerName="payload-pre-processing" containerID="cri-o://1499fdccf2872bebf8c4621a7f4a5e92498809c53cfc13cc97e299b02d01595e" gracePeriod=25 Oct 30 21:32:07.700311 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:07.700274 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz"] Oct 30 21:32:07.700861 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:07.700615 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" podUID="29d99320-f1a9-4178-97f6-dc0a088aec0b" containerName="istio-proxy" containerID="cri-o://6d17cc35eab33959c958b60194c8ccfafc9e267fb5df3c96d68a0aa503c6f2ab" gracePeriod=30 Oct 30 21:32:10.155706 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:10.155667 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4"] Oct 30 21:32:10.156172 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:10.155957 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" podUID="5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" containerName="istio-proxy" containerID="cri-o://f211bbb00eb69edfd5e13e120e59d30023b45474ed54186c45e88950c04458c3" gracePeriod=30 Oct 30 21:32:12.996595 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:12.996563 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:32:13.017402 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.017370 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-workload-socket\") pod \"29d99320-f1a9-4178-97f6-dc0a088aec0b\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " Oct 30 21:32:13.017601 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.017481 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-data\") pod \"29d99320-f1a9-4178-97f6-dc0a088aec0b\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " Oct 30 21:32:13.017601 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.017593 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-podinfo\") pod \"29d99320-f1a9-4178-97f6-dc0a088aec0b\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " Oct 30 21:32:13.017707 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.017684 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-token\" (UniqueName: \"kubernetes.io/projected/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-token\") pod \"29d99320-f1a9-4178-97f6-dc0a088aec0b\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " Oct 30 21:32:13.017750 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.017712 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-credential-socket\") pod \"29d99320-f1a9-4178-97f6-dc0a088aec0b\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " Oct 30 21:32:13.017818 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.017783 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29d99320-f1a9-4178-97f6-dc0a088aec0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29d99320-f1a9-4178-97f6-dc0a088aec0b-istiod-ca-cert\") pod \"29d99320-f1a9-4178-97f6-dc0a088aec0b\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " Oct 30 21:32:13.017871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.017834 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-envoy\") pod \"29d99320-f1a9-4178-97f6-dc0a088aec0b\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " Oct 30 21:32:13.017871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.017865 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29d99320-f1a9-4178-97f6-dc0a088aec0b-kube-api-access-djlz6\" (UniqueName: \"kubernetes.io/projected/29d99320-f1a9-4178-97f6-dc0a088aec0b-kube-api-access-djlz6\") pod \"29d99320-f1a9-4178-97f6-dc0a088aec0b\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " Oct 30 21:32:13.017954 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.017894 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-workload-certs\") pod \"29d99320-f1a9-4178-97f6-dc0a088aec0b\" (UID: \"29d99320-f1a9-4178-97f6-dc0a088aec0b\") " Oct 30 21:32:13.018524 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.018497 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-data" pod "29d99320-f1a9-4178-97f6-dc0a088aec0b" (UID: "29d99320-f1a9-4178-97f6-dc0a088aec0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:13.020464 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.018887 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29d99320-f1a9-4178-97f6-dc0a088aec0b-istiod-ca-cert" pod "29d99320-f1a9-4178-97f6-dc0a088aec0b" (UID: "29d99320-f1a9-4178-97f6-dc0a088aec0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:13.020617 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.019087 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-credential-socket" pod "29d99320-f1a9-4178-97f6-dc0a088aec0b" (UID: "29d99320-f1a9-4178-97f6-dc0a088aec0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:13.020719 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.019253 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-workload-socket" pod "29d99320-f1a9-4178-97f6-dc0a088aec0b" (UID: "29d99320-f1a9-4178-97f6-dc0a088aec0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:13.020824 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.019351 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-workload-certs" pod "29d99320-f1a9-4178-97f6-dc0a088aec0b" (UID: "29d99320-f1a9-4178-97f6-dc0a088aec0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:13.020824 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.020658 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-token" pod "29d99320-f1a9-4178-97f6-dc0a088aec0b" (UID: "29d99320-f1a9-4178-97f6-dc0a088aec0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:13.022113 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.022081 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-envoy" pod "29d99320-f1a9-4178-97f6-dc0a088aec0b" (UID: "29d99320-f1a9-4178-97f6-dc0a088aec0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:13.022487 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.022460 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29d99320-f1a9-4178-97f6-dc0a088aec0b-kube-api-access-djlz6" pod "29d99320-f1a9-4178-97f6-dc0a088aec0b" (UID: "29d99320-f1a9-4178-97f6-dc0a088aec0b"). InnerVolumeSpecName "kube-api-access-djlz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:13.023001 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.022976 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-podinfo" pod "29d99320-f1a9-4178-97f6-dc0a088aec0b" (UID: "29d99320-f1a9-4178-97f6-dc0a088aec0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:32:13.119359 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.119320 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:13.119359 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.119351 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:13.119359 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.119365 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:13.119619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.119381 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:13.119619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.119393 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29d99320-f1a9-4178-97f6-dc0a088aec0b-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:13.119619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.119405 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:13.119619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.119416 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djlz6\" (UniqueName: \"kubernetes.io/projected/29d99320-f1a9-4178-97f6-dc0a088aec0b-kube-api-access-djlz6\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:13.119619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.119428 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:13.119619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.119441 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29d99320-f1a9-4178-97f6-dc0a088aec0b-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:13.359238 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.359150 4848 generic.go:356] "Generic (PLEG): container finished" podID="29d99320-f1a9-4178-97f6-dc0a088aec0b" containerID="6d17cc35eab33959c958b60194c8ccfafc9e267fb5df3c96d68a0aa503c6f2ab" exitCode=0 Oct 30 21:32:13.359238 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.359213 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" Oct 30 21:32:13.359459 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.359215 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" event={"ID":"29d99320-f1a9-4178-97f6-dc0a088aec0b","Type":"ContainerDied","Data":"6d17cc35eab33959c958b60194c8ccfafc9e267fb5df3c96d68a0aa503c6f2ab"} Oct 30 21:32:13.359459 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.359335 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz" event={"ID":"29d99320-f1a9-4178-97f6-dc0a088aec0b","Type":"ContainerDied","Data":"b5b2c2f72705d16b890685f7993a209a5b848fd055f8bdb25bf1436ba1b80687"} Oct 30 21:32:13.359459 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.359356 4848 scope.go:122] "RemoveContainer" containerID="6d17cc35eab33959c958b60194c8ccfafc9e267fb5df3c96d68a0aa503c6f2ab" Oct 30 21:32:13.370977 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.370957 4848 scope.go:122] "RemoveContainer" containerID="599c164945c503ee6b12beb029a0e13cfc01e722e92460d0a2d32d4c42e01dab" Oct 30 21:32:13.380839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.380813 4848 scope.go:122] "RemoveContainer" containerID="6d17cc35eab33959c958b60194c8ccfafc9e267fb5df3c96d68a0aa503c6f2ab" Oct 30 21:32:13.381113 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:32:13.381096 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d17cc35eab33959c958b60194c8ccfafc9e267fb5df3c96d68a0aa503c6f2ab\": container with ID starting with 6d17cc35eab33959c958b60194c8ccfafc9e267fb5df3c96d68a0aa503c6f2ab not found: ID does not exist" containerID="6d17cc35eab33959c958b60194c8ccfafc9e267fb5df3c96d68a0aa503c6f2ab" Oct 30 21:32:13.381186 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.381123 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d17cc35eab33959c958b60194c8ccfafc9e267fb5df3c96d68a0aa503c6f2ab"} err="failed to get container status \"6d17cc35eab33959c958b60194c8ccfafc9e267fb5df3c96d68a0aa503c6f2ab\": rpc error: code = NotFound desc = could not find container \"6d17cc35eab33959c958b60194c8ccfafc9e267fb5df3c96d68a0aa503c6f2ab\": container with ID starting with 6d17cc35eab33959c958b60194c8ccfafc9e267fb5df3c96d68a0aa503c6f2ab not found: ID does not exist" Oct 30 21:32:13.381186 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.381139 4848 scope.go:122] "RemoveContainer" containerID="599c164945c503ee6b12beb029a0e13cfc01e722e92460d0a2d32d4c42e01dab" Oct 30 21:32:13.381399 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:32:13.381380 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599c164945c503ee6b12beb029a0e13cfc01e722e92460d0a2d32d4c42e01dab\": container with ID starting with 599c164945c503ee6b12beb029a0e13cfc01e722e92460d0a2d32d4c42e01dab not found: ID does not exist" containerID="599c164945c503ee6b12beb029a0e13cfc01e722e92460d0a2d32d4c42e01dab" Oct 30 21:32:13.381447 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.381406 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599c164945c503ee6b12beb029a0e13cfc01e722e92460d0a2d32d4c42e01dab"} err="failed to get container status \"599c164945c503ee6b12beb029a0e13cfc01e722e92460d0a2d32d4c42e01dab\": rpc error: code = NotFound desc = could not find container \"599c164945c503ee6b12beb029a0e13cfc01e722e92460d0a2d32d4c42e01dab\": container with ID starting with 599c164945c503ee6b12beb029a0e13cfc01e722e92460d0a2d32d4c42e01dab not found: ID does not exist" Oct 30 21:32:13.386558 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.386534 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz"] Oct 30 21:32:13.390258 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:13.390236 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-f147de-openshift-default-756bddc4df-sb6rz"] Oct 30 21:32:14.934294 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:14.934263 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29d99320-f1a9-4178-97f6-dc0a088aec0b" path="/var/lib/kubelet/pods/29d99320-f1a9-4178-97f6-dc0a088aec0b/volumes" Oct 30 21:32:15.283034 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.283000 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7" podUID="fa84afb2-c58a-4ac6-be54-bdfef411fd2b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:32:15.389039 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.389006 4848 generic.go:356] "Generic (PLEG): container finished" podID="5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" containerID="f211bbb00eb69edfd5e13e120e59d30023b45474ed54186c45e88950c04458c3" exitCode=0 Oct 30 21:32:15.389330 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.389119 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" event={"ID":"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d","Type":"ContainerDied","Data":"f211bbb00eb69edfd5e13e120e59d30023b45474ed54186c45e88950c04458c3"} Oct 30 21:32:15.389330 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.389168 4848 scope.go:122] "RemoveContainer" containerID="00f4dc6a43893ba52fa8978eedb79498e9abbf613e28ca9223114d0bed393d67" Oct 30 21:32:15.436065 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.436042 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:32:15.549314 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.549228 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-credential-socket\") pod \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " Oct 30 21:32:15.549314 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.549300 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-podinfo\") pod \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " Oct 30 21:32:15.549557 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.549456 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-kube-api-access-tdh44\" (UniqueName: \"kubernetes.io/projected/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-kube-api-access-tdh44\") pod \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " Oct 30 21:32:15.549557 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.549470 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-credential-socket" pod "5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" (UID: "5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:15.549557 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.549488 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-token\" (UniqueName: \"kubernetes.io/projected/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-token\") pod \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " Oct 30 21:32:15.549727 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.549619 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-workload-socket\") pod \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " Oct 30 21:32:15.549727 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.549682 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istiod-ca-cert\") pod \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " Oct 30 21:32:15.549887 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.549813 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-envoy\") pod \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " Oct 30 21:32:15.549887 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.549859 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-data\") pod \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " Oct 30 21:32:15.550002 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.549895 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-workload-certs\") pod \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\" (UID: \"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d\") " Oct 30 21:32:15.550002 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.549960 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-workload-socket" pod "5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" (UID: "5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:15.550192 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.550168 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-data" pod "5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" (UID: "5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:15.550401 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.550381 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-workload-certs" pod "5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" (UID: "5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:15.550472 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.550400 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istiod-ca-cert" pod "5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" (UID: "5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:15.550472 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.550428 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:15.550472 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.550451 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:15.550472 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.550467 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:15.551753 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.551728 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-token" pod "5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" (UID: "5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:15.551968 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.551947 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-podinfo" pod "5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" (UID: "5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:32:15.552044 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.551965 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-kube-api-access-tdh44" pod "5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" (UID: "5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d"). InnerVolumeSpecName "kube-api-access-tdh44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:15.552044 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.551982 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-envoy" pod "5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" (UID: "5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:15.651855 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.651823 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:15.651855 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.651852 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:15.651855 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.651861 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:15.652147 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.651870 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdh44\" (UniqueName: \"kubernetes.io/projected/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-kube-api-access-tdh44\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:15.652147 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.651879 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:15.652147 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.651888 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:15.888293 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.888224 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq"] Oct 30 21:32:15.888581 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:15.888553 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" podUID="797a8c75-fb91-4d33-8833-e062f5cd01f0" containerName="maas-api" containerID="cri-o://acaf7f0db5243bad928953775fd3c2cc3c1d2016da5566d3973ab49f9f22c672" gracePeriod=30 Oct 30 21:32:16.270609 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.270580 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" Oct 30 21:32:16.366618 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.365998 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/797a8c75-fb91-4d33-8833-e062f5cd01f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/797a8c75-fb91-4d33-8833-e062f5cd01f0-metrics-certs\") pod \"797a8c75-fb91-4d33-8833-e062f5cd01f0\" (UID: \"797a8c75-fb91-4d33-8833-e062f5cd01f0\") " Oct 30 21:32:16.366618 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.366273 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/797a8c75-fb91-4d33-8833-e062f5cd01f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/797a8c75-fb91-4d33-8833-e062f5cd01f0-maas-api-tls\") pod \"797a8c75-fb91-4d33-8833-e062f5cd01f0\" (UID: \"797a8c75-fb91-4d33-8833-e062f5cd01f0\") " Oct 30 21:32:16.366618 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.366319 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/797a8c75-fb91-4d33-8833-e062f5cd01f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/797a8c75-fb91-4d33-8833-e062f5cd01f0-openshift-service-ca\") pod \"797a8c75-fb91-4d33-8833-e062f5cd01f0\" (UID: \"797a8c75-fb91-4d33-8833-e062f5cd01f0\") " Oct 30 21:32:16.366618 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.366387 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/797a8c75-fb91-4d33-8833-e062f5cd01f0-kube-api-access-m62ft\" (UniqueName: \"kubernetes.io/projected/797a8c75-fb91-4d33-8833-e062f5cd01f0-kube-api-access-m62ft\") pod \"797a8c75-fb91-4d33-8833-e062f5cd01f0\" (UID: \"797a8c75-fb91-4d33-8833-e062f5cd01f0\") " Oct 30 21:32:16.367021 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.366754 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/797a8c75-fb91-4d33-8833-e062f5cd01f0-openshift-service-ca" pod "797a8c75-fb91-4d33-8833-e062f5cd01f0" (UID: "797a8c75-fb91-4d33-8833-e062f5cd01f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:16.378233 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.367209 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/797a8c75-fb91-4d33-8833-e062f5cd01f0-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:16.378233 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.372273 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/797a8c75-fb91-4d33-8833-e062f5cd01f0-kube-api-access-m62ft" pod "797a8c75-fb91-4d33-8833-e062f5cd01f0" (UID: "797a8c75-fb91-4d33-8833-e062f5cd01f0"). InnerVolumeSpecName "kube-api-access-m62ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:16.378233 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.372452 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/797a8c75-fb91-4d33-8833-e062f5cd01f0-metrics-certs" pod "797a8c75-fb91-4d33-8833-e062f5cd01f0" (UID: "797a8c75-fb91-4d33-8833-e062f5cd01f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:16.378233 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.374372 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/797a8c75-fb91-4d33-8833-e062f5cd01f0-maas-api-tls" pod "797a8c75-fb91-4d33-8833-e062f5cd01f0" (UID: "797a8c75-fb91-4d33-8833-e062f5cd01f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:16.400112 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.400076 4848 generic.go:356] "Generic (PLEG): container finished" podID="797a8c75-fb91-4d33-8833-e062f5cd01f0" containerID="acaf7f0db5243bad928953775fd3c2cc3c1d2016da5566d3973ab49f9f22c672" exitCode=0 Oct 30 21:32:16.400350 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.400147 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" Oct 30 21:32:16.400350 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.400161 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" event={"ID":"797a8c75-fb91-4d33-8833-e062f5cd01f0","Type":"ContainerDied","Data":"acaf7f0db5243bad928953775fd3c2cc3c1d2016da5566d3973ab49f9f22c672"} Oct 30 21:32:16.400350 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.400219 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq" event={"ID":"797a8c75-fb91-4d33-8833-e062f5cd01f0","Type":"ContainerDied","Data":"25df88775572e82745cca7d929b507fe0990fc18121706da9ea2dcc8f6a15ca8"} Oct 30 21:32:16.400350 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.400247 4848 scope.go:122] "RemoveContainer" containerID="acaf7f0db5243bad928953775fd3c2cc3c1d2016da5566d3973ab49f9f22c672" Oct 30 21:32:16.413049 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.412995 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" event={"ID":"5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d","Type":"ContainerDied","Data":"1449687a00a94e1cd9deaaf0aeab9ae25d0a25ea75295c27991e8174721712fd"} Oct 30 21:32:16.413233 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.413147 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4" Oct 30 21:32:16.424026 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.424006 4848 scope.go:122] "RemoveContainer" containerID="acaf7f0db5243bad928953775fd3c2cc3c1d2016da5566d3973ab49f9f22c672" Oct 30 21:32:16.424365 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:32:16.424335 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acaf7f0db5243bad928953775fd3c2cc3c1d2016da5566d3973ab49f9f22c672\": container with ID starting with acaf7f0db5243bad928953775fd3c2cc3c1d2016da5566d3973ab49f9f22c672 not found: ID does not exist" containerID="acaf7f0db5243bad928953775fd3c2cc3c1d2016da5566d3973ab49f9f22c672" Oct 30 21:32:16.424464 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.424378 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acaf7f0db5243bad928953775fd3c2cc3c1d2016da5566d3973ab49f9f22c672"} err="failed to get container status \"acaf7f0db5243bad928953775fd3c2cc3c1d2016da5566d3973ab49f9f22c672\": rpc error: code = NotFound desc = could not find container \"acaf7f0db5243bad928953775fd3c2cc3c1d2016da5566d3973ab49f9f22c672\": container with ID starting with acaf7f0db5243bad928953775fd3c2cc3c1d2016da5566d3973ab49f9f22c672 not found: ID does not exist" Oct 30 21:32:16.424464 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.424403 4848 scope.go:122] "RemoveContainer" containerID="f211bbb00eb69edfd5e13e120e59d30023b45474ed54186c45e88950c04458c3" Oct 30 21:32:16.433341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.433311 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq"] Oct 30 21:32:16.436546 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.436525 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-a50675-95bfff7dc-pdhjq"] Oct 30 21:32:16.447366 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.447312 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4"] Oct 30 21:32:16.452414 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.452392 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0c4052-openshift-default-7d5b5958dc-2rll4"] Oct 30 21:32:16.468312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.468288 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m62ft\" (UniqueName: \"kubernetes.io/projected/797a8c75-fb91-4d33-8833-e062f5cd01f0-kube-api-access-m62ft\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:16.468312 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.468313 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/797a8c75-fb91-4d33-8833-e062f5cd01f0-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:16.468461 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.468326 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/797a8c75-fb91-4d33-8833-e062f5cd01f0-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:16.933331 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.933295 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" path="/var/lib/kubelet/pods/5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d/volumes" Oct 30 21:32:16.933892 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:16.933870 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="797a8c75-fb91-4d33-8833-e062f5cd01f0" path="/var/lib/kubelet/pods/797a8c75-fb91-4d33-8833-e062f5cd01f0/volumes" Oct 30 21:32:20.880975 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:20.880942 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx"] Oct 30 21:32:22.010060 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.010026 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck"] Oct 30 21:32:22.010583 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.010306 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" podUID="72874e47-9bf8-4fcd-98b8-4c51b124244e" containerName="maas-api" containerID="cri-o://48b710eb9b0aa7f5696acb05f9d6a1c124f6cf1ef5d15921a3c5a29b15651c69" gracePeriod=30 Oct 30 21:32:22.337688 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.337660 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" Oct 30 21:32:22.446281 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.446247 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72874e47-9bf8-4fcd-98b8-4c51b124244e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72874e47-9bf8-4fcd-98b8-4c51b124244e-metrics-certs\") pod \"72874e47-9bf8-4fcd-98b8-4c51b124244e\" (UID: \"72874e47-9bf8-4fcd-98b8-4c51b124244e\") " Oct 30 21:32:22.446489 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.446376 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72874e47-9bf8-4fcd-98b8-4c51b124244e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72874e47-9bf8-4fcd-98b8-4c51b124244e-maas-api-tls\") pod \"72874e47-9bf8-4fcd-98b8-4c51b124244e\" (UID: \"72874e47-9bf8-4fcd-98b8-4c51b124244e\") " Oct 30 21:32:22.446489 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.446428 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72874e47-9bf8-4fcd-98b8-4c51b124244e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72874e47-9bf8-4fcd-98b8-4c51b124244e-openshift-service-ca\") pod \"72874e47-9bf8-4fcd-98b8-4c51b124244e\" (UID: \"72874e47-9bf8-4fcd-98b8-4c51b124244e\") " Oct 30 21:32:22.446489 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.446461 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72874e47-9bf8-4fcd-98b8-4c51b124244e-kube-api-access-79mqc\" (UniqueName: \"kubernetes.io/projected/72874e47-9bf8-4fcd-98b8-4c51b124244e-kube-api-access-79mqc\") pod \"72874e47-9bf8-4fcd-98b8-4c51b124244e\" (UID: \"72874e47-9bf8-4fcd-98b8-4c51b124244e\") " Oct 30 21:32:22.446942 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.446888 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72874e47-9bf8-4fcd-98b8-4c51b124244e-openshift-service-ca" pod "72874e47-9bf8-4fcd-98b8-4c51b124244e" (UID: "72874e47-9bf8-4fcd-98b8-4c51b124244e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:22.449141 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.449113 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72874e47-9bf8-4fcd-98b8-4c51b124244e-maas-api-tls" pod "72874e47-9bf8-4fcd-98b8-4c51b124244e" (UID: "72874e47-9bf8-4fcd-98b8-4c51b124244e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:22.449254 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.449137 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72874e47-9bf8-4fcd-98b8-4c51b124244e-metrics-certs" pod "72874e47-9bf8-4fcd-98b8-4c51b124244e" (UID: "72874e47-9bf8-4fcd-98b8-4c51b124244e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:22.449310 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.449265 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72874e47-9bf8-4fcd-98b8-4c51b124244e-kube-api-access-79mqc" pod "72874e47-9bf8-4fcd-98b8-4c51b124244e" (UID: "72874e47-9bf8-4fcd-98b8-4c51b124244e"). InnerVolumeSpecName "kube-api-access-79mqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:22.452134 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.452108 4848 generic.go:356] "Generic (PLEG): container finished" podID="72874e47-9bf8-4fcd-98b8-4c51b124244e" containerID="48b710eb9b0aa7f5696acb05f9d6a1c124f6cf1ef5d15921a3c5a29b15651c69" exitCode=0 Oct 30 21:32:22.452250 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.452159 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" Oct 30 21:32:22.452250 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.452187 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" event={"ID":"72874e47-9bf8-4fcd-98b8-4c51b124244e","Type":"ContainerDied","Data":"48b710eb9b0aa7f5696acb05f9d6a1c124f6cf1ef5d15921a3c5a29b15651c69"} Oct 30 21:32:22.452250 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.452237 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck" event={"ID":"72874e47-9bf8-4fcd-98b8-4c51b124244e","Type":"ContainerDied","Data":"086569921a70544de0730eb75466158a13d1f869e464438b614db0d22830c9fb"} Oct 30 21:32:22.452419 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.452259 4848 scope.go:122] "RemoveContainer" containerID="48b710eb9b0aa7f5696acb05f9d6a1c124f6cf1ef5d15921a3c5a29b15651c69" Oct 30 21:32:22.474982 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.474961 4848 scope.go:122] "RemoveContainer" containerID="48b710eb9b0aa7f5696acb05f9d6a1c124f6cf1ef5d15921a3c5a29b15651c69" Oct 30 21:32:22.475314 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:32:22.475289 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48b710eb9b0aa7f5696acb05f9d6a1c124f6cf1ef5d15921a3c5a29b15651c69\": container with ID starting with 48b710eb9b0aa7f5696acb05f9d6a1c124f6cf1ef5d15921a3c5a29b15651c69 not found: ID does not exist" containerID="48b710eb9b0aa7f5696acb05f9d6a1c124f6cf1ef5d15921a3c5a29b15651c69" Oct 30 21:32:22.475416 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.475320 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b710eb9b0aa7f5696acb05f9d6a1c124f6cf1ef5d15921a3c5a29b15651c69"} err="failed to get container status \"48b710eb9b0aa7f5696acb05f9d6a1c124f6cf1ef5d15921a3c5a29b15651c69\": rpc error: code = NotFound desc = could not find container \"48b710eb9b0aa7f5696acb05f9d6a1c124f6cf1ef5d15921a3c5a29b15651c69\": container with ID starting with 48b710eb9b0aa7f5696acb05f9d6a1c124f6cf1ef5d15921a3c5a29b15651c69 not found: ID does not exist" Oct 30 21:32:22.488387 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.488350 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck"] Oct 30 21:32:22.499063 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.499032 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-f64640-849b7fdcf4-hvlck"] Oct 30 21:32:22.530668 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:32:22.527640 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:32:22.547731 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.547687 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72874e47-9bf8-4fcd-98b8-4c51b124244e-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:22.547731 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.547731 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72874e47-9bf8-4fcd-98b8-4c51b124244e-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:22.547973 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.547745 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79mqc\" (UniqueName: \"kubernetes.io/projected/72874e47-9bf8-4fcd-98b8-4c51b124244e-kube-api-access-79mqc\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:22.547973 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.547761 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72874e47-9bf8-4fcd-98b8-4c51b124244e-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:22.932989 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:22.932955 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72874e47-9bf8-4fcd-98b8-4c51b124244e" path="/var/lib/kubelet/pods/72874e47-9bf8-4fcd-98b8-4c51b124244e/volumes" Oct 30 21:32:24.797735 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:24.797701 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b"] Oct 30 21:32:24.798265 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:24.797958 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" podUID="ebde61df-6d3a-43f2-8ab0-e61c220a5259" containerName="istio-proxy" containerID="cri-o://4210d4095055540cf3dd7f289005f56248921fca8eca50fc7a657243968ab9c5" gracePeriod=30 Oct 30 21:32:25.138563 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.138487 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc"] Oct 30 21:32:25.138792 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.138718 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" podUID="ee6e41be-d07f-4346-a538-8bd375b21b9b" containerName="maas-api" containerID="cri-o://0d065528b126fa7258fdd06d699390fac017475bfc83ba41ee54ef8ece493bee" gracePeriod=30 Oct 30 21:32:25.282965 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.282926 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7" podUID="fa84afb2-c58a-4ac6-be54-bdfef411fd2b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:32:25.459206 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.459173 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" Oct 30 21:32:25.472711 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.472678 4848 generic.go:356] "Generic (PLEG): container finished" podID="ee6e41be-d07f-4346-a538-8bd375b21b9b" containerID="0d065528b126fa7258fdd06d699390fac017475bfc83ba41ee54ef8ece493bee" exitCode=0 Oct 30 21:32:25.472923 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.472741 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" Oct 30 21:32:25.472923 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.472759 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" event={"ID":"ee6e41be-d07f-4346-a538-8bd375b21b9b","Type":"ContainerDied","Data":"0d065528b126fa7258fdd06d699390fac017475bfc83ba41ee54ef8ece493bee"} Oct 30 21:32:25.472923 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.472845 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc" event={"ID":"ee6e41be-d07f-4346-a538-8bd375b21b9b","Type":"ContainerDied","Data":"200c4e145cc2eed05ed03f6e6e1501a8e2412efa847367e4f25640f65cfa45f7"} Oct 30 21:32:25.472923 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.472875 4848 scope.go:122] "RemoveContainer" containerID="0d065528b126fa7258fdd06d699390fac017475bfc83ba41ee54ef8ece493bee" Oct 30 21:32:25.485370 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.485344 4848 scope.go:122] "RemoveContainer" containerID="0d065528b126fa7258fdd06d699390fac017475bfc83ba41ee54ef8ece493bee" Oct 30 21:32:25.485710 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:32:25.485678 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d065528b126fa7258fdd06d699390fac017475bfc83ba41ee54ef8ece493bee\": container with ID starting with 0d065528b126fa7258fdd06d699390fac017475bfc83ba41ee54ef8ece493bee not found: ID does not exist" containerID="0d065528b126fa7258fdd06d699390fac017475bfc83ba41ee54ef8ece493bee" Oct 30 21:32:25.485880 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.485719 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d065528b126fa7258fdd06d699390fac017475bfc83ba41ee54ef8ece493bee"} err="failed to get container status \"0d065528b126fa7258fdd06d699390fac017475bfc83ba41ee54ef8ece493bee\": rpc error: code = NotFound desc = could not find container \"0d065528b126fa7258fdd06d699390fac017475bfc83ba41ee54ef8ece493bee\": container with ID starting with 0d065528b126fa7258fdd06d699390fac017475bfc83ba41ee54ef8ece493bee not found: ID does not exist" Oct 30 21:32:25.588418 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.588389 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee6e41be-d07f-4346-a538-8bd375b21b9b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee6e41be-d07f-4346-a538-8bd375b21b9b-openshift-service-ca\") pod \"ee6e41be-d07f-4346-a538-8bd375b21b9b\" (UID: \"ee6e41be-d07f-4346-a538-8bd375b21b9b\") " Oct 30 21:32:25.588590 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.588487 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee6e41be-d07f-4346-a538-8bd375b21b9b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee6e41be-d07f-4346-a538-8bd375b21b9b-metrics-certs\") pod \"ee6e41be-d07f-4346-a538-8bd375b21b9b\" (UID: \"ee6e41be-d07f-4346-a538-8bd375b21b9b\") " Oct 30 21:32:25.588590 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.588533 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee6e41be-d07f-4346-a538-8bd375b21b9b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee6e41be-d07f-4346-a538-8bd375b21b9b-maas-api-tls\") pod \"ee6e41be-d07f-4346-a538-8bd375b21b9b\" (UID: \"ee6e41be-d07f-4346-a538-8bd375b21b9b\") " Oct 30 21:32:25.588665 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.588591 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee6e41be-d07f-4346-a538-8bd375b21b9b-kube-api-access-vkvqf\" (UniqueName: \"kubernetes.io/projected/ee6e41be-d07f-4346-a538-8bd375b21b9b-kube-api-access-vkvqf\") pod \"ee6e41be-d07f-4346-a538-8bd375b21b9b\" (UID: \"ee6e41be-d07f-4346-a538-8bd375b21b9b\") " Oct 30 21:32:25.588796 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.588758 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee6e41be-d07f-4346-a538-8bd375b21b9b-openshift-service-ca" pod "ee6e41be-d07f-4346-a538-8bd375b21b9b" (UID: "ee6e41be-d07f-4346-a538-8bd375b21b9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:25.589120 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.589098 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee6e41be-d07f-4346-a538-8bd375b21b9b-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:25.590693 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.590665 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee6e41be-d07f-4346-a538-8bd375b21b9b-kube-api-access-vkvqf" pod "ee6e41be-d07f-4346-a538-8bd375b21b9b" (UID: "ee6e41be-d07f-4346-a538-8bd375b21b9b"). InnerVolumeSpecName "kube-api-access-vkvqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:25.590693 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.590687 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee6e41be-d07f-4346-a538-8bd375b21b9b-maas-api-tls" pod "ee6e41be-d07f-4346-a538-8bd375b21b9b" (UID: "ee6e41be-d07f-4346-a538-8bd375b21b9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:25.590898 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.590661 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee6e41be-d07f-4346-a538-8bd375b21b9b-metrics-certs" pod "ee6e41be-d07f-4346-a538-8bd375b21b9b" (UID: "ee6e41be-d07f-4346-a538-8bd375b21b9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:25.690670 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.690590 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee6e41be-d07f-4346-a538-8bd375b21b9b-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:25.690670 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.690617 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee6e41be-d07f-4346-a538-8bd375b21b9b-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:25.690670 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.690628 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkvqf\" (UniqueName: \"kubernetes.io/projected/ee6e41be-d07f-4346-a538-8bd375b21b9b-kube-api-access-vkvqf\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:25.804116 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.804089 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc"] Oct 30 21:32:25.812115 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.812089 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f8e2c3-75fddcb898-d8tnc"] Oct 30 21:32:25.834385 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.834363 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7" Oct 30 21:32:25.881849 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.881813 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx" podUID="69dbc4b2-1701-4de6-a810-795807a04c62" containerName="payload-pre-processing" containerID="cri-o://dc47f4171a9f956d2ee9c563e8c72648c37f29ce9be1a301aee7c2e7299370fa" gracePeriod=25 Oct 30 21:32:25.994814 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.994720 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa84afb2-c58a-4ac6-be54-bdfef411fd2b-kube-api-access-mt5bq\" (UniqueName: \"kubernetes.io/projected/fa84afb2-c58a-4ac6-be54-bdfef411fd2b-kube-api-access-mt5bq\") pod \"fa84afb2-c58a-4ac6-be54-bdfef411fd2b\" (UID: \"fa84afb2-c58a-4ac6-be54-bdfef411fd2b\") " Oct 30 21:32:25.994963 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.994896 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa84afb2-c58a-4ac6-be54-bdfef411fd2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa84afb2-c58a-4ac6-be54-bdfef411fd2b-plugins-config-volume\") pod \"fa84afb2-c58a-4ac6-be54-bdfef411fd2b\" (UID: \"fa84afb2-c58a-4ac6-be54-bdfef411fd2b\") " Oct 30 21:32:25.995242 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.995219 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa84afb2-c58a-4ac6-be54-bdfef411fd2b-plugins-config-volume" pod "fa84afb2-c58a-4ac6-be54-bdfef411fd2b" (UID: "fa84afb2-c58a-4ac6-be54-bdfef411fd2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:25.996821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:25.996790 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa84afb2-c58a-4ac6-be54-bdfef411fd2b-kube-api-access-mt5bq" pod "fa84afb2-c58a-4ac6-be54-bdfef411fd2b" (UID: "fa84afb2-c58a-4ac6-be54-bdfef411fd2b"). InnerVolumeSpecName "kube-api-access-mt5bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:26.096987 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:26.096949 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt5bq\" (UniqueName: \"kubernetes.io/projected/fa84afb2-c58a-4ac6-be54-bdfef411fd2b-kube-api-access-mt5bq\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:26.096987 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:26.096977 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa84afb2-c58a-4ac6-be54-bdfef411fd2b-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:26.287326 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:26.287243 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6"] Oct 30 21:32:26.287488 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:26.287432 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" podUID="9e35d342-d32b-4cbb-8249-6a4c12ce82cd" containerName="istio-proxy" containerID="cri-o://9ab7378278af6a7d204ba1663483f335916eca23b93912d2cb3aeb55b8ae367d" gracePeriod=30 Oct 30 21:32:26.479192 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:26.479160 4848 generic.go:356] "Generic (PLEG): container finished" podID="fa84afb2-c58a-4ac6-be54-bdfef411fd2b" containerID="1499fdccf2872bebf8c4621a7f4a5e92498809c53cfc13cc97e299b02d01595e" exitCode=0 Oct 30 21:32:26.479402 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:26.479204 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7" Oct 30 21:32:26.479402 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:26.479241 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7" event={"ID":"fa84afb2-c58a-4ac6-be54-bdfef411fd2b","Type":"ContainerDied","Data":"1499fdccf2872bebf8c4621a7f4a5e92498809c53cfc13cc97e299b02d01595e"} Oct 30 21:32:26.479402 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:26.479294 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7" event={"ID":"fa84afb2-c58a-4ac6-be54-bdfef411fd2b","Type":"ContainerDied","Data":"7e1db3b97c060a66e32e36f13ead93bcdc77935e95cbe3431e451b746df4a678"} Oct 30 21:32:26.479402 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:26.479320 4848 scope.go:122] "RemoveContainer" containerID="1499fdccf2872bebf8c4621a7f4a5e92498809c53cfc13cc97e299b02d01595e" Oct 30 21:32:26.489583 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:26.489558 4848 scope.go:122] "RemoveContainer" containerID="1499fdccf2872bebf8c4621a7f4a5e92498809c53cfc13cc97e299b02d01595e" Oct 30 21:32:26.489853 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:32:26.489835 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1499fdccf2872bebf8c4621a7f4a5e92498809c53cfc13cc97e299b02d01595e\": container with ID starting with 1499fdccf2872bebf8c4621a7f4a5e92498809c53cfc13cc97e299b02d01595e not found: ID does not exist" containerID="1499fdccf2872bebf8c4621a7f4a5e92498809c53cfc13cc97e299b02d01595e" Oct 30 21:32:26.489929 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:26.489862 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1499fdccf2872bebf8c4621a7f4a5e92498809c53cfc13cc97e299b02d01595e"} err="failed to get container status \"1499fdccf2872bebf8c4621a7f4a5e92498809c53cfc13cc97e299b02d01595e\": rpc error: code = NotFound desc = could not find container \"1499fdccf2872bebf8c4621a7f4a5e92498809c53cfc13cc97e299b02d01595e\": container with ID starting with 1499fdccf2872bebf8c4621a7f4a5e92498809c53cfc13cc97e299b02d01595e not found: ID does not exist" Oct 30 21:32:26.515901 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:26.515874 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7"] Oct 30 21:32:26.521905 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:26.521879 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0c4052-6f5875db4c-9jxk7"] Oct 30 21:32:26.932284 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:26.932252 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee6e41be-d07f-4346-a538-8bd375b21b9b" path="/var/lib/kubelet/pods/ee6e41be-d07f-4346-a538-8bd375b21b9b/volumes" Oct 30 21:32:26.932724 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:26.932708 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa84afb2-c58a-4ac6-be54-bdfef411fd2b" path="/var/lib/kubelet/pods/fa84afb2-c58a-4ac6-be54-bdfef411fd2b/volumes" Oct 30 21:32:30.076059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.075993 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:32:30.246749 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.246710 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-envoy\") pod \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " Oct 30 21:32:30.246955 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.246756 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-podinfo\") pod \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " Oct 30 21:32:30.246955 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.246869 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebde61df-6d3a-43f2-8ab0-e61c220a5259-kube-api-access-r68kd\" (UniqueName: \"kubernetes.io/projected/ebde61df-6d3a-43f2-8ab0-e61c220a5259-kube-api-access-r68kd\") pod \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " Oct 30 21:32:30.246955 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.246897 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-data\") pod \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " Oct 30 21:32:30.247143 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.246962 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-workload-socket\") pod \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " Oct 30 21:32:30.247143 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.247036 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-token\" (UniqueName: \"kubernetes.io/projected/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-token\") pod \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " Oct 30 21:32:30.247143 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.247127 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-credential-socket\") pod \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " Oct 30 21:32:30.247314 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.247171 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istiod-ca-cert\") pod \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " Oct 30 21:32:30.247314 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.247232 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-workload-certs\") pod \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\" (UID: \"ebde61df-6d3a-43f2-8ab0-e61c220a5259\") " Oct 30 21:32:30.247314 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.247229 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-data" pod "ebde61df-6d3a-43f2-8ab0-e61c220a5259" (UID: "ebde61df-6d3a-43f2-8ab0-e61c220a5259"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:30.247314 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.247275 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-workload-socket" pod "ebde61df-6d3a-43f2-8ab0-e61c220a5259" (UID: "ebde61df-6d3a-43f2-8ab0-e61c220a5259"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:30.247554 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.247425 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-credential-socket" pod "ebde61df-6d3a-43f2-8ab0-e61c220a5259" (UID: "ebde61df-6d3a-43f2-8ab0-e61c220a5259"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:30.247554 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.247490 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-workload-certs" pod "ebde61df-6d3a-43f2-8ab0-e61c220a5259" (UID: "ebde61df-6d3a-43f2-8ab0-e61c220a5259"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:30.247663 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.247600 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istiod-ca-cert" pod "ebde61df-6d3a-43f2-8ab0-e61c220a5259" (UID: "ebde61df-6d3a-43f2-8ab0-e61c220a5259"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:30.247882 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.247860 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:30.247937 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.247890 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:30.247937 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.247908 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:30.247937 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.247923 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:30.248034 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.247937 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:30.249169 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.249140 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-podinfo" pod "ebde61df-6d3a-43f2-8ab0-e61c220a5259" (UID: "ebde61df-6d3a-43f2-8ab0-e61c220a5259"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:32:30.249270 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.249242 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-token" pod "ebde61df-6d3a-43f2-8ab0-e61c220a5259" (UID: "ebde61df-6d3a-43f2-8ab0-e61c220a5259"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:30.249440 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.249412 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-envoy" pod "ebde61df-6d3a-43f2-8ab0-e61c220a5259" (UID: "ebde61df-6d3a-43f2-8ab0-e61c220a5259"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:30.249674 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.249656 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebde61df-6d3a-43f2-8ab0-e61c220a5259-kube-api-access-r68kd" pod "ebde61df-6d3a-43f2-8ab0-e61c220a5259" (UID: "ebde61df-6d3a-43f2-8ab0-e61c220a5259"). InnerVolumeSpecName "kube-api-access-r68kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:30.349547 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.349464 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:30.349547 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.349498 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:30.349547 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.349509 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r68kd\" (UniqueName: \"kubernetes.io/projected/ebde61df-6d3a-43f2-8ab0-e61c220a5259-kube-api-access-r68kd\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:30.349547 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.349519 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebde61df-6d3a-43f2-8ab0-e61c220a5259-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:30.425078 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.425010 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr"] Oct 30 21:32:30.436576 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.436543 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt"] Oct 30 21:32:30.519293 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.519265 4848 generic.go:356] "Generic (PLEG): container finished" podID="ebde61df-6d3a-43f2-8ab0-e61c220a5259" containerID="4210d4095055540cf3dd7f289005f56248921fca8eca50fc7a657243968ab9c5" exitCode=0 Oct 30 21:32:30.519465 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.519318 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" Oct 30 21:32:30.519465 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.519346 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" event={"ID":"ebde61df-6d3a-43f2-8ab0-e61c220a5259","Type":"ContainerDied","Data":"4210d4095055540cf3dd7f289005f56248921fca8eca50fc7a657243968ab9c5"} Oct 30 21:32:30.519465 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.519398 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b" event={"ID":"ebde61df-6d3a-43f2-8ab0-e61c220a5259","Type":"ContainerDied","Data":"cd68a58a2d2dbe715ec128423843ff35b0d31039de30cacc15d85c253d039eaa"} Oct 30 21:32:30.519465 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.519420 4848 scope.go:122] "RemoveContainer" containerID="4210d4095055540cf3dd7f289005f56248921fca8eca50fc7a657243968ab9c5" Oct 30 21:32:30.529965 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.529950 4848 scope.go:122] "RemoveContainer" containerID="b5a6137eac4ae7e616ed1b7897b1d7a9b81bad785ab7e96e43ee25468eeaf6bf" Oct 30 21:32:30.539261 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.539245 4848 scope.go:122] "RemoveContainer" containerID="4210d4095055540cf3dd7f289005f56248921fca8eca50fc7a657243968ab9c5" Oct 30 21:32:30.539523 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:32:30.539500 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4210d4095055540cf3dd7f289005f56248921fca8eca50fc7a657243968ab9c5\": container with ID starting with 4210d4095055540cf3dd7f289005f56248921fca8eca50fc7a657243968ab9c5 not found: ID does not exist" containerID="4210d4095055540cf3dd7f289005f56248921fca8eca50fc7a657243968ab9c5" Oct 30 21:32:30.539632 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.539528 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4210d4095055540cf3dd7f289005f56248921fca8eca50fc7a657243968ab9c5"} err="failed to get container status \"4210d4095055540cf3dd7f289005f56248921fca8eca50fc7a657243968ab9c5\": rpc error: code = NotFound desc = could not find container \"4210d4095055540cf3dd7f289005f56248921fca8eca50fc7a657243968ab9c5\": container with ID starting with 4210d4095055540cf3dd7f289005f56248921fca8eca50fc7a657243968ab9c5 not found: ID does not exist" Oct 30 21:32:30.539632 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.539545 4848 scope.go:122] "RemoveContainer" containerID="b5a6137eac4ae7e616ed1b7897b1d7a9b81bad785ab7e96e43ee25468eeaf6bf" Oct 30 21:32:30.539858 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:32:30.539839 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5a6137eac4ae7e616ed1b7897b1d7a9b81bad785ab7e96e43ee25468eeaf6bf\": container with ID starting with b5a6137eac4ae7e616ed1b7897b1d7a9b81bad785ab7e96e43ee25468eeaf6bf not found: ID does not exist" containerID="b5a6137eac4ae7e616ed1b7897b1d7a9b81bad785ab7e96e43ee25468eeaf6bf" Oct 30 21:32:30.539921 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.539875 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5a6137eac4ae7e616ed1b7897b1d7a9b81bad785ab7e96e43ee25468eeaf6bf"} err="failed to get container status \"b5a6137eac4ae7e616ed1b7897b1d7a9b81bad785ab7e96e43ee25468eeaf6bf\": rpc error: code = NotFound desc = could not find container \"b5a6137eac4ae7e616ed1b7897b1d7a9b81bad785ab7e96e43ee25468eeaf6bf\": container with ID starting with b5a6137eac4ae7e616ed1b7897b1d7a9b81bad785ab7e96e43ee25468eeaf6bf not found: ID does not exist" Oct 30 21:32:30.549252 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.549233 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b"] Oct 30 21:32:30.560132 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.560104 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-a50675-openshift-default-85dcc5678f-bd64b"] Oct 30 21:32:30.665206 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.665122 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7"] Oct 30 21:32:30.665718 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.665688 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" podUID="7f1686ee-2109-4bfa-9964-cc40955a35b9" containerName="istio-proxy" containerID="cri-o://608b801315fffc5cd5f7ee45d8912e02add5a0fa8241292356be24351cb215d4" gracePeriod=30 Oct 30 21:32:30.932142 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:30.932104 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebde61df-6d3a-43f2-8ab0-e61c220a5259" path="/var/lib/kubelet/pods/ebde61df-6d3a-43f2-8ab0-e61c220a5259/volumes" Oct 30 21:32:31.537835 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.536961 4848 generic.go:356] "Generic (PLEG): container finished" podID="9e35d342-d32b-4cbb-8249-6a4c12ce82cd" containerID="9ab7378278af6a7d204ba1663483f335916eca23b93912d2cb3aeb55b8ae367d" exitCode=0 Oct 30 21:32:31.537835 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.537033 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" event={"ID":"9e35d342-d32b-4cbb-8249-6a4c12ce82cd","Type":"ContainerDied","Data":"9ab7378278af6a7d204ba1663483f335916eca23b93912d2cb3aeb55b8ae367d"} Oct 30 21:32:31.619688 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.619660 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:32:31.767472 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.767433 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istiod-ca-cert\") pod \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " Oct 30 21:32:31.767682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.767480 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-credential-socket\") pod \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " Oct 30 21:32:31.767682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.767504 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-envoy\") pod \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " Oct 30 21:32:31.767682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.767526 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-kube-api-access-g4htf\" (UniqueName: \"kubernetes.io/projected/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-kube-api-access-g4htf\") pod \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " Oct 30 21:32:31.767682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.767585 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-token\" (UniqueName: \"kubernetes.io/projected/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-token\") pod \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " Oct 30 21:32:31.767682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.767614 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-workload-certs\") pod \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " Oct 30 21:32:31.768024 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.767860 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-podinfo\") pod \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " Oct 30 21:32:31.768024 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.767897 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-workload-socket\") pod \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " Oct 30 21:32:31.768024 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.767952 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-data\") pod \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\" (UID: \"9e35d342-d32b-4cbb-8249-6a4c12ce82cd\") " Oct 30 21:32:31.768206 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.768015 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-workload-certs" pod "9e35d342-d32b-4cbb-8249-6a4c12ce82cd" (UID: "9e35d342-d32b-4cbb-8249-6a4c12ce82cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:31.768206 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.768020 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istiod-ca-cert" pod "9e35d342-d32b-4cbb-8249-6a4c12ce82cd" (UID: "9e35d342-d32b-4cbb-8249-6a4c12ce82cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:31.768321 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.768296 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-workload-socket" pod "9e35d342-d32b-4cbb-8249-6a4c12ce82cd" (UID: "9e35d342-d32b-4cbb-8249-6a4c12ce82cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:31.768489 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.768440 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:31.768612 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.768493 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:31.768612 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.768508 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:31.768612 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.768504 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-data" pod "9e35d342-d32b-4cbb-8249-6a4c12ce82cd" (UID: "9e35d342-d32b-4cbb-8249-6a4c12ce82cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:31.768895 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.768654 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-credential-socket" pod "9e35d342-d32b-4cbb-8249-6a4c12ce82cd" (UID: "9e35d342-d32b-4cbb-8249-6a4c12ce82cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:31.770509 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.770478 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-token" pod "9e35d342-d32b-4cbb-8249-6a4c12ce82cd" (UID: "9e35d342-d32b-4cbb-8249-6a4c12ce82cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:31.770605 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.770481 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-podinfo" pod "9e35d342-d32b-4cbb-8249-6a4c12ce82cd" (UID: "9e35d342-d32b-4cbb-8249-6a4c12ce82cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:32:31.770605 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.770484 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-envoy" pod "9e35d342-d32b-4cbb-8249-6a4c12ce82cd" (UID: "9e35d342-d32b-4cbb-8249-6a4c12ce82cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:31.770729 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.770650 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-kube-api-access-g4htf" pod "9e35d342-d32b-4cbb-8249-6a4c12ce82cd" (UID: "9e35d342-d32b-4cbb-8249-6a4c12ce82cd"). InnerVolumeSpecName "kube-api-access-g4htf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:31.869520 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.869434 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:31.869520 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.869467 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:31.869520 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.869480 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:31.869520 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.869494 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:31.869520 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.869507 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4htf\" (UniqueName: \"kubernetes.io/projected/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-kube-api-access-g4htf\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:31.869520 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:31.869521 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e35d342-d32b-4cbb-8249-6a4c12ce82cd-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:32.545021 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:32.544989 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" event={"ID":"9e35d342-d32b-4cbb-8249-6a4c12ce82cd","Type":"ContainerDied","Data":"04445245a5cfef6d346b6ad08890d48be57e8dec27c1ce4a9a502b2e3cf04f74"} Oct 30 21:32:32.545470 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:32.545052 4848 scope.go:122] "RemoveContainer" containerID="9ab7378278af6a7d204ba1663483f335916eca23b93912d2cb3aeb55b8ae367d" Oct 30 21:32:32.545470 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:32.544999 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6" Oct 30 21:32:32.570337 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:32.570310 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6"] Oct 30 21:32:32.574950 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:32.574929 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-qbpx6"] Oct 30 21:32:32.931432 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:32.931402 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e35d342-d32b-4cbb-8249-6a4c12ce82cd" path="/var/lib/kubelet/pods/9e35d342-d32b-4cbb-8249-6a4c12ce82cd/volumes" Oct 30 21:32:33.543761 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:33.543721 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r"] Oct 30 21:32:33.544068 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:33.544016 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" podUID="4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" containerName="istio-proxy" containerID="cri-o://0fbc51e19561e98c35558c4d36169bb7cb5674d2695abe00c7f2fb69668b08e9" gracePeriod=30 Oct 30 21:32:35.772295 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:35.772256 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx" podUID="69dbc4b2-1701-4de6-a810-795807a04c62" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:32:35.837749 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:35.837712 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86"] Oct 30 21:32:35.983158 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:35.983135 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:32:36.123858 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.123819 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f1686ee-2109-4bfa-9964-cc40955a35b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f1686ee-2109-4bfa-9964-cc40955a35b9-istiod-ca-cert\") pod \"7f1686ee-2109-4bfa-9964-cc40955a35b9\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " Oct 30 21:32:36.124076 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.123927 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-podinfo\") pod \"7f1686ee-2109-4bfa-9964-cc40955a35b9\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " Oct 30 21:32:36.124076 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.123959 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-data\") pod \"7f1686ee-2109-4bfa-9964-cc40955a35b9\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " Oct 30 21:32:36.124076 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.124011 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-credential-socket\") pod \"7f1686ee-2109-4bfa-9964-cc40955a35b9\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " Oct 30 21:32:36.124320 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.124086 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-workload-socket\") pod \"7f1686ee-2109-4bfa-9964-cc40955a35b9\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " Oct 30 21:32:36.124320 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.124121 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-token\" (UniqueName: \"kubernetes.io/projected/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-token\") pod \"7f1686ee-2109-4bfa-9964-cc40955a35b9\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " Oct 30 21:32:36.124320 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.124173 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-envoy\") pod \"7f1686ee-2109-4bfa-9964-cc40955a35b9\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " Oct 30 21:32:36.124320 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.124209 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-workload-certs\") pod \"7f1686ee-2109-4bfa-9964-cc40955a35b9\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " Oct 30 21:32:36.124320 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.124218 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f1686ee-2109-4bfa-9964-cc40955a35b9-istiod-ca-cert" pod "7f1686ee-2109-4bfa-9964-cc40955a35b9" (UID: "7f1686ee-2109-4bfa-9964-cc40955a35b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:36.124320 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.124247 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f1686ee-2109-4bfa-9964-cc40955a35b9-kube-api-access-rlv6n\" (UniqueName: \"kubernetes.io/projected/7f1686ee-2109-4bfa-9964-cc40955a35b9-kube-api-access-rlv6n\") pod \"7f1686ee-2109-4bfa-9964-cc40955a35b9\" (UID: \"7f1686ee-2109-4bfa-9964-cc40955a35b9\") " Oct 30 21:32:36.124640 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.124446 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-workload-socket" pod "7f1686ee-2109-4bfa-9964-cc40955a35b9" (UID: "7f1686ee-2109-4bfa-9964-cc40955a35b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:36.124640 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.124575 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-data" pod "7f1686ee-2109-4bfa-9964-cc40955a35b9" (UID: "7f1686ee-2109-4bfa-9964-cc40955a35b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:36.124738 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.124717 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:36.124738 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.124735 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f1686ee-2109-4bfa-9964-cc40955a35b9-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:36.124879 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.124748 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:36.124984 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.124952 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-credential-socket" pod "7f1686ee-2109-4bfa-9964-cc40955a35b9" (UID: "7f1686ee-2109-4bfa-9964-cc40955a35b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:36.125330 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.125305 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-workload-certs" pod "7f1686ee-2109-4bfa-9964-cc40955a35b9" (UID: "7f1686ee-2109-4bfa-9964-cc40955a35b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:36.127501 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.127474 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f1686ee-2109-4bfa-9964-cc40955a35b9-kube-api-access-rlv6n" pod "7f1686ee-2109-4bfa-9964-cc40955a35b9" (UID: "7f1686ee-2109-4bfa-9964-cc40955a35b9"). InnerVolumeSpecName "kube-api-access-rlv6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:36.127631 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.127573 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-podinfo" pod "7f1686ee-2109-4bfa-9964-cc40955a35b9" (UID: "7f1686ee-2109-4bfa-9964-cc40955a35b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:32:36.127901 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.127881 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-envoy" pod "7f1686ee-2109-4bfa-9964-cc40955a35b9" (UID: "7f1686ee-2109-4bfa-9964-cc40955a35b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:36.128735 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.128709 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-token" pod "7f1686ee-2109-4bfa-9964-cc40955a35b9" (UID: "7f1686ee-2109-4bfa-9964-cc40955a35b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:36.168211 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.168183 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4"] Oct 30 21:32:36.188101 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.187948 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp"] Oct 30 21:32:36.225853 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.225744 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlv6n\" (UniqueName: \"kubernetes.io/projected/7f1686ee-2109-4bfa-9964-cc40955a35b9-kube-api-access-rlv6n\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:36.225853 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.225788 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:36.225853 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.225804 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:36.225853 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.225815 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:36.225853 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.225828 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:36.225853 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.225840 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f1686ee-2109-4bfa-9964-cc40955a35b9-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:36.573192 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.573095 4848 generic.go:356] "Generic (PLEG): container finished" podID="7f1686ee-2109-4bfa-9964-cc40955a35b9" containerID="608b801315fffc5cd5f7ee45d8912e02add5a0fa8241292356be24351cb215d4" exitCode=0 Oct 30 21:32:36.573192 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.573153 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" Oct 30 21:32:36.573420 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.573268 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" event={"ID":"7f1686ee-2109-4bfa-9964-cc40955a35b9","Type":"ContainerDied","Data":"608b801315fffc5cd5f7ee45d8912e02add5a0fa8241292356be24351cb215d4"} Oct 30 21:32:36.573420 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.573319 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7" event={"ID":"7f1686ee-2109-4bfa-9964-cc40955a35b9","Type":"ContainerDied","Data":"ce5b94087b4c788148082cedab2637342b93abc1c060ced091e7f188ac1a9e35"} Oct 30 21:32:36.573420 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.573349 4848 scope.go:122] "RemoveContainer" containerID="608b801315fffc5cd5f7ee45d8912e02add5a0fa8241292356be24351cb215d4" Oct 30 21:32:36.585679 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.585651 4848 scope.go:122] "RemoveContainer" containerID="794183a3dee72e74d092a786187b0bc7b8acf59556b005b53f7359353a405e26" Oct 30 21:32:36.597820 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.597735 4848 scope.go:122] "RemoveContainer" containerID="608b801315fffc5cd5f7ee45d8912e02add5a0fa8241292356be24351cb215d4" Oct 30 21:32:36.598153 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:32:36.598131 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"608b801315fffc5cd5f7ee45d8912e02add5a0fa8241292356be24351cb215d4\": container with ID starting with 608b801315fffc5cd5f7ee45d8912e02add5a0fa8241292356be24351cb215d4 not found: ID does not exist" containerID="608b801315fffc5cd5f7ee45d8912e02add5a0fa8241292356be24351cb215d4" Oct 30 21:32:36.598263 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.598166 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608b801315fffc5cd5f7ee45d8912e02add5a0fa8241292356be24351cb215d4"} err="failed to get container status \"608b801315fffc5cd5f7ee45d8912e02add5a0fa8241292356be24351cb215d4\": rpc error: code = NotFound desc = could not find container \"608b801315fffc5cd5f7ee45d8912e02add5a0fa8241292356be24351cb215d4\": container with ID starting with 608b801315fffc5cd5f7ee45d8912e02add5a0fa8241292356be24351cb215d4 not found: ID does not exist" Oct 30 21:32:36.598263 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.598191 4848 scope.go:122] "RemoveContainer" containerID="794183a3dee72e74d092a786187b0bc7b8acf59556b005b53f7359353a405e26" Oct 30 21:32:36.598496 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:32:36.598479 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"794183a3dee72e74d092a786187b0bc7b8acf59556b005b53f7359353a405e26\": container with ID starting with 794183a3dee72e74d092a786187b0bc7b8acf59556b005b53f7359353a405e26 not found: ID does not exist" containerID="794183a3dee72e74d092a786187b0bc7b8acf59556b005b53f7359353a405e26" Oct 30 21:32:36.598563 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.598505 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"794183a3dee72e74d092a786187b0bc7b8acf59556b005b53f7359353a405e26"} err="failed to get container status \"794183a3dee72e74d092a786187b0bc7b8acf59556b005b53f7359353a405e26\": rpc error: code = NotFound desc = could not find container \"794183a3dee72e74d092a786187b0bc7b8acf59556b005b53f7359353a405e26\": container with ID starting with 794183a3dee72e74d092a786187b0bc7b8acf59556b005b53f7359353a405e26 not found: ID does not exist" Oct 30 21:32:36.607208 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.607182 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7"] Oct 30 21:32:36.613255 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.613204 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f64640-openshift-default-d945bfdb7-wr5k7"] Oct 30 21:32:36.931720 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:36.931690 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f1686ee-2109-4bfa-9964-cc40955a35b9" path="/var/lib/kubelet/pods/7f1686ee-2109-4bfa-9964-cc40955a35b9/volumes" Oct 30 21:32:38.289224 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.289192 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx"] Oct 30 21:32:38.289699 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.289484 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" podUID="9f7e1ecd-c614-4687-96e3-631e5716cf83" containerName="maas-api" containerID="cri-o://2257b757aa8b729d0f2132a0da342937e3b711820c1b9b2a7644e8dd32f9b92e" gracePeriod=30 Oct 30 21:32:38.587643 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.587611 4848 generic.go:356] "Generic (PLEG): container finished" podID="9f7e1ecd-c614-4687-96e3-631e5716cf83" containerID="2257b757aa8b729d0f2132a0da342937e3b711820c1b9b2a7644e8dd32f9b92e" exitCode=0 Oct 30 21:32:38.587779 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.587677 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" event={"ID":"9f7e1ecd-c614-4687-96e3-631e5716cf83","Type":"ContainerDied","Data":"2257b757aa8b729d0f2132a0da342937e3b711820c1b9b2a7644e8dd32f9b92e"} Oct 30 21:32:38.587779 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.587728 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" event={"ID":"9f7e1ecd-c614-4687-96e3-631e5716cf83","Type":"ContainerDied","Data":"fe56079c3e08a4c67b8443ee9e66f6f9330da33a7c897efb7c810b33cdeec081"} Oct 30 21:32:38.587779 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.587752 4848 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe56079c3e08a4c67b8443ee9e66f6f9330da33a7c897efb7c810b33cdeec081" Oct 30 21:32:38.640238 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.640213 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" Oct 30 21:32:38.757998 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.757964 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f7e1ecd-c614-4687-96e3-631e5716cf83-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f7e1ecd-c614-4687-96e3-631e5716cf83-openshift-service-ca\") pod \"9f7e1ecd-c614-4687-96e3-631e5716cf83\" (UID: \"9f7e1ecd-c614-4687-96e3-631e5716cf83\") " Oct 30 21:32:38.758138 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.758033 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f7e1ecd-c614-4687-96e3-631e5716cf83-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f7e1ecd-c614-4687-96e3-631e5716cf83-metrics-certs\") pod \"9f7e1ecd-c614-4687-96e3-631e5716cf83\" (UID: \"9f7e1ecd-c614-4687-96e3-631e5716cf83\") " Oct 30 21:32:38.758138 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.758082 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f7e1ecd-c614-4687-96e3-631e5716cf83-kube-api-access-dl489\" (UniqueName: \"kubernetes.io/projected/9f7e1ecd-c614-4687-96e3-631e5716cf83-kube-api-access-dl489\") pod \"9f7e1ecd-c614-4687-96e3-631e5716cf83\" (UID: \"9f7e1ecd-c614-4687-96e3-631e5716cf83\") " Oct 30 21:32:38.758410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.758210 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f7e1ecd-c614-4687-96e3-631e5716cf83-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f7e1ecd-c614-4687-96e3-631e5716cf83-maas-api-tls\") pod \"9f7e1ecd-c614-4687-96e3-631e5716cf83\" (UID: \"9f7e1ecd-c614-4687-96e3-631e5716cf83\") " Oct 30 21:32:38.758410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.758398 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f7e1ecd-c614-4687-96e3-631e5716cf83-openshift-service-ca" pod "9f7e1ecd-c614-4687-96e3-631e5716cf83" (UID: "9f7e1ecd-c614-4687-96e3-631e5716cf83"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:38.759471 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.759449 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f7e1ecd-c614-4687-96e3-631e5716cf83-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.760971 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.760937 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f7e1ecd-c614-4687-96e3-631e5716cf83-maas-api-tls" pod "9f7e1ecd-c614-4687-96e3-631e5716cf83" (UID: "9f7e1ecd-c614-4687-96e3-631e5716cf83"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:38.762027 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.762002 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f7e1ecd-c614-4687-96e3-631e5716cf83-metrics-certs" pod "9f7e1ecd-c614-4687-96e3-631e5716cf83" (UID: "9f7e1ecd-c614-4687-96e3-631e5716cf83"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:38.765558 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.765524 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f7e1ecd-c614-4687-96e3-631e5716cf83-kube-api-access-dl489" pod "9f7e1ecd-c614-4687-96e3-631e5716cf83" (UID: "9f7e1ecd-c614-4687-96e3-631e5716cf83"). InnerVolumeSpecName "kube-api-access-dl489". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:38.827690 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.827665 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:32:38.860301 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.860262 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f7e1ecd-c614-4687-96e3-631e5716cf83-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.860301 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.860301 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl489\" (UniqueName: \"kubernetes.io/projected/9f7e1ecd-c614-4687-96e3-631e5716cf83-kube-api-access-dl489\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.860576 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.860318 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f7e1ecd-c614-4687-96e3-631e5716cf83-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.961309 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.961281 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-podinfo\") pod \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " Oct 30 21:32:38.961502 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.961377 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-credential-socket\") pod \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " Oct 30 21:32:38.961502 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.961446 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-workload-socket\") pod \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " Oct 30 21:32:38.961502 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.961492 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-envoy\") pod \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " Oct 30 21:32:38.961689 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.961539 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istiod-ca-cert\") pod \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " Oct 30 21:32:38.961689 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.961597 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-kube-api-access-p659r\" (UniqueName: \"kubernetes.io/projected/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-kube-api-access-p659r\") pod \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " Oct 30 21:32:38.961689 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.961637 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-data\") pod \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " Oct 30 21:32:38.961689 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.961669 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-workload-certs\") pod \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " Oct 30 21:32:38.961942 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.961715 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-token\" (UniqueName: \"kubernetes.io/projected/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-token\") pod \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\" (UID: \"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e\") " Oct 30 21:32:38.961942 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.961744 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-workload-socket" pod "4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" (UID: "4c4c39fa-0000-40ad-8f24-53ae45fe0f7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:38.961942 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.961778 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-credential-socket" pod "4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" (UID: "4c4c39fa-0000-40ad-8f24-53ae45fe0f7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:38.962334 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.962145 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-data" pod "4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" (UID: "4c4c39fa-0000-40ad-8f24-53ae45fe0f7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:38.962334 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.962193 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.962334 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.962214 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:38.962334 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.962271 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-workload-certs" pod "4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" (UID: "4c4c39fa-0000-40ad-8f24-53ae45fe0f7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:38.962334 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.962308 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istiod-ca-cert" pod "4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" (UID: "4c4c39fa-0000-40ad-8f24-53ae45fe0f7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:38.964055 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.964030 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-envoy" pod "4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" (UID: "4c4c39fa-0000-40ad-8f24-53ae45fe0f7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:38.964166 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.964055 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-kube-api-access-p659r" pod "4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" (UID: "4c4c39fa-0000-40ad-8f24-53ae45fe0f7e"). InnerVolumeSpecName "kube-api-access-p659r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:38.964166 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.964041 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-podinfo" pod "4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" (UID: "4c4c39fa-0000-40ad-8f24-53ae45fe0f7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:32:38.964704 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:38.964678 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-token" pod "4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" (UID: "4c4c39fa-0000-40ad-8f24-53ae45fe0f7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:39.063062 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.063027 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:39.063062 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.063055 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:39.063062 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.063064 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:39.063305 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.063073 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p659r\" (UniqueName: \"kubernetes.io/projected/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-kube-api-access-p659r\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:39.063305 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.063084 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:39.063305 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.063094 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:39.063305 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.063102 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:39.595132 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.595049 4848 generic.go:356] "Generic (PLEG): container finished" podID="4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" containerID="0fbc51e19561e98c35558c4d36169bb7cb5674d2695abe00c7f2fb69668b08e9" exitCode=0 Oct 30 21:32:39.595132 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.595107 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" Oct 30 21:32:39.595578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.595129 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" event={"ID":"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e","Type":"ContainerDied","Data":"0fbc51e19561e98c35558c4d36169bb7cb5674d2695abe00c7f2fb69668b08e9"} Oct 30 21:32:39.595578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.595181 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r" event={"ID":"4c4c39fa-0000-40ad-8f24-53ae45fe0f7e","Type":"ContainerDied","Data":"1592592fe7eaf0ce741fba5f7db124d59ef0d60648b687f331fa9713cf10055e"} Oct 30 21:32:39.595578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.595204 4848 scope.go:122] "RemoveContainer" containerID="0fbc51e19561e98c35558c4d36169bb7cb5674d2695abe00c7f2fb69668b08e9" Oct 30 21:32:39.595578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.595362 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx" Oct 30 21:32:39.625895 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.625871 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx"] Oct 30 21:32:39.628172 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.628150 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-53aa47-689b9bd556-gkmwx"] Oct 30 21:32:39.630127 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.630113 4848 scope.go:122] "RemoveContainer" containerID="1f17dcfe5a7265ddd079ef9f424a4032a4bf5d488d203b679d6958bc3d3bfe76" Oct 30 21:32:39.640241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.640225 4848 scope.go:122] "RemoveContainer" containerID="0fbc51e19561e98c35558c4d36169bb7cb5674d2695abe00c7f2fb69668b08e9" Oct 30 21:32:39.640545 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:32:39.640526 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fbc51e19561e98c35558c4d36169bb7cb5674d2695abe00c7f2fb69668b08e9\": container with ID starting with 0fbc51e19561e98c35558c4d36169bb7cb5674d2695abe00c7f2fb69668b08e9 not found: ID does not exist" containerID="0fbc51e19561e98c35558c4d36169bb7cb5674d2695abe00c7f2fb69668b08e9" Oct 30 21:32:39.640594 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.640553 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fbc51e19561e98c35558c4d36169bb7cb5674d2695abe00c7f2fb69668b08e9"} err="failed to get container status \"0fbc51e19561e98c35558c4d36169bb7cb5674d2695abe00c7f2fb69668b08e9\": rpc error: code = NotFound desc = could not find container \"0fbc51e19561e98c35558c4d36169bb7cb5674d2695abe00c7f2fb69668b08e9\": container with ID starting with 0fbc51e19561e98c35558c4d36169bb7cb5674d2695abe00c7f2fb69668b08e9 not found: ID does not exist" Oct 30 21:32:39.640594 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.640570 4848 scope.go:122] "RemoveContainer" containerID="1f17dcfe5a7265ddd079ef9f424a4032a4bf5d488d203b679d6958bc3d3bfe76" Oct 30 21:32:39.640829 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:32:39.640806 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f17dcfe5a7265ddd079ef9f424a4032a4bf5d488d203b679d6958bc3d3bfe76\": container with ID starting with 1f17dcfe5a7265ddd079ef9f424a4032a4bf5d488d203b679d6958bc3d3bfe76 not found: ID does not exist" containerID="1f17dcfe5a7265ddd079ef9f424a4032a4bf5d488d203b679d6958bc3d3bfe76" Oct 30 21:32:39.640906 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.640837 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f17dcfe5a7265ddd079ef9f424a4032a4bf5d488d203b679d6958bc3d3bfe76"} err="failed to get container status \"1f17dcfe5a7265ddd079ef9f424a4032a4bf5d488d203b679d6958bc3d3bfe76\": rpc error: code = NotFound desc = could not find container \"1f17dcfe5a7265ddd079ef9f424a4032a4bf5d488d203b679d6958bc3d3bfe76\": container with ID starting with 1f17dcfe5a7265ddd079ef9f424a4032a4bf5d488d203b679d6958bc3d3bfe76 not found: ID does not exist" Oct 30 21:32:39.643697 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.643674 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r"] Oct 30 21:32:39.650866 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:39.650848 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f8e2c3-openshift-default-9b5c4b479-vkf7r"] Oct 30 21:32:40.846077 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:40.846009 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86" podUID="b7f2555a-89d9-4d58-89d5-f076b60af465" containerName="payload-pre-processing" containerID="cri-o://272be5d67b61f9b848032198b237badf12d5794f07a997f8245a6ddc98fcdac5" gracePeriod=25 Oct 30 21:32:40.931288 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:40.931256 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" path="/var/lib/kubelet/pods/4c4c39fa-0000-40ad-8f24-53ae45fe0f7e/volumes" Oct 30 21:32:40.931695 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:40.931682 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f7e1ecd-c614-4687-96e3-631e5716cf83" path="/var/lib/kubelet/pods/9f7e1ecd-c614-4687-96e3-631e5716cf83/volumes" Oct 30 21:32:41.477652 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:41.477619 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d"] Oct 30 21:32:41.483744 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:41.483718 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr"] Oct 30 21:32:45.446743 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.446655 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" containerID="cri-o://bd747dd58ce1109aea46f23c1fb0d6057bbd979a8a4743420c626b871af41b73" gracePeriod=45 Oct 30 21:32:45.461707 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.461671 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" containerID="cri-o://e366d5861fd5412c5af70e1fc81901b6afde74226dc13beda28a50f5d8e3ae20" gracePeriod=45 Oct 30 21:32:45.637949 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.633554 4848 generic.go:356] "Generic (PLEG): container finished" podID="9184c819-a720-46b2-992b-65a4087be757" containerID="bd747dd58ce1109aea46f23c1fb0d6057bbd979a8a4743420c626b871af41b73" exitCode=0 Oct 30 21:32:45.637949 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.633720 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" event={"ID":"9184c819-a720-46b2-992b-65a4087be757","Type":"ContainerDied","Data":"bd747dd58ce1109aea46f23c1fb0d6057bbd979a8a4743420c626b871af41b73"} Oct 30 21:32:45.637949 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.635845 4848 generic.go:356] "Generic (PLEG): container finished" podID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerID="e366d5861fd5412c5af70e1fc81901b6afde74226dc13beda28a50f5d8e3ae20" exitCode=0 Oct 30 21:32:45.637949 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.635919 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" event={"ID":"fa47948a-4865-45d9-a687-fc7fbcc3a6a8","Type":"ContainerDied","Data":"e366d5861fd5412c5af70e1fc81901b6afde74226dc13beda28a50f5d8e3ae20"} Oct 30 21:32:45.741923 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.741893 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 21:32:45.752938 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.752906 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-model-cache\") pod \"9184c819-a720-46b2-992b-65a4087be757\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " Oct 30 21:32:45.753087 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.752963 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-tmp-dir\") pod \"9184c819-a720-46b2-992b-65a4087be757\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " Oct 30 21:32:45.753087 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.753013 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-home\" (UniqueName: \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-home\") pod \"9184c819-a720-46b2-992b-65a4087be757\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " Oct 30 21:32:45.753087 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.753054 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9184c819-a720-46b2-992b-65a4087be757-tls-certs\" (UniqueName: \"kubernetes.io/secret/9184c819-a720-46b2-992b-65a4087be757-tls-certs\") pod \"9184c819-a720-46b2-992b-65a4087be757\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " Oct 30 21:32:45.753283 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.753228 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9184c819-a720-46b2-992b-65a4087be757-kube-api-access-mg5hl\" (UniqueName: \"kubernetes.io/projected/9184c819-a720-46b2-992b-65a4087be757-kube-api-access-mg5hl\") pod \"9184c819-a720-46b2-992b-65a4087be757\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " Oct 30 21:32:45.753283 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.753245 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-model-cache" pod "9184c819-a720-46b2-992b-65a4087be757" (UID: "9184c819-a720-46b2-992b-65a4087be757"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:45.753283 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.753273 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-dshm\") pod \"9184c819-a720-46b2-992b-65a4087be757\" (UID: \"9184c819-a720-46b2-992b-65a4087be757\") " Oct 30 21:32:45.753461 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.753389 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-tmp-dir" pod "9184c819-a720-46b2-992b-65a4087be757" (UID: "9184c819-a720-46b2-992b-65a4087be757"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:45.753528 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.753464 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-home" pod "9184c819-a720-46b2-992b-65a4087be757" (UID: "9184c819-a720-46b2-992b-65a4087be757"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:45.753813 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.753736 4848 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-model-cache\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:45.753813 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.753776 4848 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-tmp-dir\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:45.753813 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.753793 4848 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-home\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:45.755863 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.755838 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-dshm" pod "9184c819-a720-46b2-992b-65a4087be757" (UID: "9184c819-a720-46b2-992b-65a4087be757"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:45.755863 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.755852 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9184c819-a720-46b2-992b-65a4087be757-kube-api-access-mg5hl" pod "9184c819-a720-46b2-992b-65a4087be757" (UID: "9184c819-a720-46b2-992b-65a4087be757"). InnerVolumeSpecName "kube-api-access-mg5hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:45.756008 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.755976 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9184c819-a720-46b2-992b-65a4087be757-tls-certs" pod "9184c819-a720-46b2-992b-65a4087be757" (UID: "9184c819-a720-46b2-992b-65a4087be757"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:45.765385 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.765363 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 21:32:45.772246 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.772216 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx" podUID="69dbc4b2-1701-4de6-a810-795807a04c62" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:32:45.854952 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.854923 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-tls-certs\") pod \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " Oct 30 21:32:45.855171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.854969 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-home\" (UniqueName: \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-home\") pod \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " Oct 30 21:32:45.855171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.855054 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-model-cache\") pod \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " Oct 30 21:32:45.855171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.855128 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-kube-api-access-pqrjg\" (UniqueName: \"kubernetes.io/projected/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-kube-api-access-pqrjg\") pod \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " Oct 30 21:32:45.855348 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.855173 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-tmp-dir\") pod \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " Oct 30 21:32:45.855348 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.855261 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-dshm\") pod \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\" (UID: \"fa47948a-4865-45d9-a687-fc7fbcc3a6a8\") " Oct 30 21:32:45.855348 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.855269 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-home" pod "fa47948a-4865-45d9-a687-fc7fbcc3a6a8" (UID: "fa47948a-4865-45d9-a687-fc7fbcc3a6a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:45.855509 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.855344 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-model-cache" pod "fa47948a-4865-45d9-a687-fc7fbcc3a6a8" (UID: "fa47948a-4865-45d9-a687-fc7fbcc3a6a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:45.855509 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.855500 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-tmp-dir" pod "fa47948a-4865-45d9-a687-fc7fbcc3a6a8" (UID: "fa47948a-4865-45d9-a687-fc7fbcc3a6a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:45.855692 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.855673 4848 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-model-cache\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:45.855783 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.855695 4848 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-tmp-dir\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:45.855783 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.855711 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg5hl\" (UniqueName: \"kubernetes.io/projected/9184c819-a720-46b2-992b-65a4087be757-kube-api-access-mg5hl\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:45.855783 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.855724 4848 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9184c819-a720-46b2-992b-65a4087be757-dshm\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:45.855783 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.855737 4848 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-home\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:45.855783 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.855752 4848 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9184c819-a720-46b2-992b-65a4087be757-tls-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:45.857231 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.857205 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-tls-certs" pod "fa47948a-4865-45d9-a687-fc7fbcc3a6a8" (UID: "fa47948a-4865-45d9-a687-fc7fbcc3a6a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:45.857368 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.857306 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-dshm" pod "fa47948a-4865-45d9-a687-fc7fbcc3a6a8" (UID: "fa47948a-4865-45d9-a687-fc7fbcc3a6a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:45.857450 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.857435 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-kube-api-access-pqrjg" pod "fa47948a-4865-45d9-a687-fc7fbcc3a6a8" (UID: "fa47948a-4865-45d9-a687-fc7fbcc3a6a8"). InnerVolumeSpecName "kube-api-access-pqrjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:45.956462 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.956432 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqrjg\" (UniqueName: \"kubernetes.io/projected/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-kube-api-access-pqrjg\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:45.956462 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.956461 4848 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-dshm\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:45.956678 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:45.956470 4848 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa47948a-4865-45d9-a687-fc7fbcc3a6a8-tls-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:46.127778 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.127747 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx" Oct 30 21:32:46.158179 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.158152 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69dbc4b2-1701-4de6-a810-795807a04c62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69dbc4b2-1701-4de6-a810-795807a04c62-plugins-config-volume\") pod \"69dbc4b2-1701-4de6-a810-795807a04c62\" (UID: \"69dbc4b2-1701-4de6-a810-795807a04c62\") " Oct 30 21:32:46.158395 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.158251 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69dbc4b2-1701-4de6-a810-795807a04c62-kube-api-access-nr2tm\" (UniqueName: \"kubernetes.io/projected/69dbc4b2-1701-4de6-a810-795807a04c62-kube-api-access-nr2tm\") pod \"69dbc4b2-1701-4de6-a810-795807a04c62\" (UID: \"69dbc4b2-1701-4de6-a810-795807a04c62\") " Oct 30 21:32:46.159542 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.159513 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69dbc4b2-1701-4de6-a810-795807a04c62-plugins-config-volume" pod "69dbc4b2-1701-4de6-a810-795807a04c62" (UID: "69dbc4b2-1701-4de6-a810-795807a04c62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:46.161064 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.161040 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69dbc4b2-1701-4de6-a810-795807a04c62-kube-api-access-nr2tm" pod "69dbc4b2-1701-4de6-a810-795807a04c62" (UID: "69dbc4b2-1701-4de6-a810-795807a04c62"). InnerVolumeSpecName "kube-api-access-nr2tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:46.260608 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.260527 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr2tm\" (UniqueName: \"kubernetes.io/projected/69dbc4b2-1701-4de6-a810-795807a04c62-kube-api-access-nr2tm\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:46.260608 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.260555 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69dbc4b2-1701-4de6-a810-795807a04c62-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:46.478785 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.478701 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d" podUID="3d385f07-6dae-4ca6-9e02-eb587a7d6374" containerName="payload-processing" containerID="cri-o://53571241600ecbf7f3e5f0e47a2a7f5173a59897a2988b637da5af2f2d94e164" gracePeriod=25 Oct 30 21:32:46.484623 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.484592 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr" podUID="6867e706-6187-4821-ae3c-141f443718a5" containerName="payload-pre-processing" containerID="cri-o://ba0c27980a493c24c11f878bce0f8fde72b2cb67b173c248d6b2b8377f93d3cd" gracePeriod=25 Oct 30 21:32:46.642694 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.642616 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" Oct 30 21:32:46.642694 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.642647 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt" event={"ID":"fa47948a-4865-45d9-a687-fc7fbcc3a6a8","Type":"ContainerDied","Data":"4c098752cbbf792ee4bdd76002e4951152b2a2e1a2dd5482f7b10b9b6560eb02"} Oct 30 21:32:46.642976 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.642695 4848 scope.go:122] "RemoveContainer" containerID="e366d5861fd5412c5af70e1fc81901b6afde74226dc13beda28a50f5d8e3ae20" Oct 30 21:32:46.644109 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.644084 4848 generic.go:356] "Generic (PLEG): container finished" podID="69dbc4b2-1701-4de6-a810-795807a04c62" containerID="dc47f4171a9f956d2ee9c563e8c72648c37f29ce9be1a301aee7c2e7299370fa" exitCode=0 Oct 30 21:32:46.644243 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.644126 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx" Oct 30 21:32:46.644243 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.644167 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx" event={"ID":"69dbc4b2-1701-4de6-a810-795807a04c62","Type":"ContainerDied","Data":"dc47f4171a9f956d2ee9c563e8c72648c37f29ce9be1a301aee7c2e7299370fa"} Oct 30 21:32:46.644243 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.644223 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx" event={"ID":"69dbc4b2-1701-4de6-a810-795807a04c62","Type":"ContainerDied","Data":"d0af641cafb21e390bae7cf6c396f422b2c831440857132a3a77a039bcabf315"} Oct 30 21:32:46.645812 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.645787 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" event={"ID":"9184c819-a720-46b2-992b-65a4087be757","Type":"ContainerDied","Data":"38d5c6b7d2c967e034e0112875f70289825a7e37f92970ca46351b508fb18ad9"} Oct 30 21:32:46.645946 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.645814 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr" Oct 30 21:32:46.653437 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.653412 4848 scope.go:122] "RemoveContainer" containerID="dc47f4171a9f956d2ee9c563e8c72648c37f29ce9be1a301aee7c2e7299370fa" Oct 30 21:32:46.662889 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.662866 4848 scope.go:122] "RemoveContainer" containerID="dc47f4171a9f956d2ee9c563e8c72648c37f29ce9be1a301aee7c2e7299370fa" Oct 30 21:32:46.663455 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:32:46.663435 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc47f4171a9f956d2ee9c563e8c72648c37f29ce9be1a301aee7c2e7299370fa\": container with ID starting with dc47f4171a9f956d2ee9c563e8c72648c37f29ce9be1a301aee7c2e7299370fa not found: ID does not exist" containerID="dc47f4171a9f956d2ee9c563e8c72648c37f29ce9be1a301aee7c2e7299370fa" Oct 30 21:32:46.663517 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.663461 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc47f4171a9f956d2ee9c563e8c72648c37f29ce9be1a301aee7c2e7299370fa"} err="failed to get container status \"dc47f4171a9f956d2ee9c563e8c72648c37f29ce9be1a301aee7c2e7299370fa\": rpc error: code = NotFound desc = could not find container \"dc47f4171a9f956d2ee9c563e8c72648c37f29ce9be1a301aee7c2e7299370fa\": container with ID starting with dc47f4171a9f956d2ee9c563e8c72648c37f29ce9be1a301aee7c2e7299370fa not found: ID does not exist" Oct 30 21:32:46.663517 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.663477 4848 scope.go:122] "RemoveContainer" containerID="bd747dd58ce1109aea46f23c1fb0d6057bbd979a8a4743420c626b871af41b73" Oct 30 21:32:46.678292 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.678271 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt"] Oct 30 21:32:46.680182 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.680163 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/simulator-w0-bc102e-kserve-798dd5948b-7tnxt"] Oct 30 21:32:46.689367 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.689344 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx"] Oct 30 21:32:46.693734 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.693714 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f8e2c3-7566d47887-ns6bx"] Oct 30 21:32:46.704858 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.704833 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr"] Oct 30 21:32:46.707480 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.707456 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-bc102e/distinct-w0-bc102e-kserve-7d4898f6d6-65rhr"] Oct 30 21:32:46.869051 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.868752 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f21776fa-f824n"] Oct 30 21:32:46.869403 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869382 4848 cpu_manager.go:414] "containerMap: removing container" podUID="f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" containerName="istio-proxy" Oct 30 21:32:46.869525 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869405 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" containerName="istio-proxy" Oct 30 21:32:46.869525 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869431 4848 cpu_manager.go:414] "containerMap: removing container" podUID="797a8c75-fb91-4d33-8833-e062f5cd01f0" containerName="maas-api" Oct 30 21:32:46.869525 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869439 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="797a8c75-fb91-4d33-8833-e062f5cd01f0" containerName="maas-api" Oct 30 21:32:46.869525 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869454 4848 cpu_manager.go:414] "containerMap: removing container" podUID="ee6e41be-d07f-4346-a538-8bd375b21b9b" containerName="maas-api" Oct 30 21:32:46.869525 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869461 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee6e41be-d07f-4346-a538-8bd375b21b9b" containerName="maas-api" Oct 30 21:32:46.869525 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869504 4848 cpu_manager.go:414] "containerMap: removing container" podUID="5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" containerName="istio-proxy" Oct 30 21:32:46.869525 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869512 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" containerName="istio-proxy" Oct 30 21:32:46.869525 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869524 4848 cpu_manager.go:414] "containerMap: removing container" podUID="4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869533 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869542 4848 cpu_manager.go:414] "containerMap: removing container" podUID="01a47632-b985-464f-8ee1-4fe60a1b3258" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869549 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a47632-b985-464f-8ee1-4fe60a1b3258" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869558 4848 cpu_manager.go:414] "containerMap: removing container" podUID="f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869565 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869575 4848 cpu_manager.go:414] "containerMap: removing container" podUID="3974a64c-bb4d-436e-bcfe-da5277f71f8c" containerName="maas-api" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869582 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3974a64c-bb4d-436e-bcfe-da5277f71f8c" containerName="maas-api" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869591 4848 cpu_manager.go:414] "containerMap: removing container" podUID="ebde61df-6d3a-43f2-8ab0-e61c220a5259" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869597 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebde61df-6d3a-43f2-8ab0-e61c220a5259" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869612 4848 cpu_manager.go:414] "containerMap: removing container" podUID="fa84afb2-c58a-4ac6-be54-bdfef411fd2b" containerName="payload-pre-processing" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869622 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa84afb2-c58a-4ac6-be54-bdfef411fd2b" containerName="payload-pre-processing" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869634 4848 cpu_manager.go:414] "containerMap: removing container" podUID="28231432-c0bf-4c3b-879e-40acaceddeec" containerName="cleanup" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869642 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28231432-c0bf-4c3b-879e-40acaceddeec" containerName="cleanup" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869651 4848 cpu_manager.go:414] "containerMap: removing container" podUID="7c946890-dd61-47f5-8c84-ef70e09b2da4" containerName="cleanup" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869657 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c946890-dd61-47f5-8c84-ef70e09b2da4" containerName="cleanup" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869670 4848 cpu_manager.go:414] "containerMap: removing container" podUID="b02fca84-f078-4d46-a842-a9b29f63eaaa" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869677 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02fca84-f078-4d46-a842-a9b29f63eaaa" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869690 4848 cpu_manager.go:414] "containerMap: removing container" podUID="691eb0e8-a109-4d75-90c5-d7a7bedd9be0" containerName="cleanup" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869697 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="691eb0e8-a109-4d75-90c5-d7a7bedd9be0" containerName="cleanup" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869727 4848 cpu_manager.go:414] "containerMap: removing container" podUID="81767769-741f-4780-93d2-32406b2a6ba6" containerName="maas-api" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869735 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81767769-741f-4780-93d2-32406b2a6ba6" containerName="maas-api" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869761 4848 cpu_manager.go:414] "containerMap: removing container" podUID="c31751bb-34fe-4e1f-8bcf-2f9b457d3379" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869807 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31751bb-34fe-4e1f-8bcf-2f9b457d3379" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869842 4848 cpu_manager.go:414] "containerMap: removing container" podUID="9f7e1ecd-c614-4687-96e3-631e5716cf83" containerName="maas-api" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869876 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f7e1ecd-c614-4687-96e3-631e5716cf83" containerName="maas-api" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869888 4848 cpu_manager.go:414] "containerMap: removing container" podUID="b02fca84-f078-4d46-a842-a9b29f63eaaa" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869896 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02fca84-f078-4d46-a842-a9b29f63eaaa" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869907 4848 cpu_manager.go:414] "containerMap: removing container" podUID="ebde61df-6d3a-43f2-8ab0-e61c220a5259" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869914 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebde61df-6d3a-43f2-8ab0-e61c220a5259" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869925 4848 cpu_manager.go:414] "containerMap: removing container" podUID="1f478d7b-4805-4559-93cb-33e622b3a537" containerName="discovery" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869931 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f478d7b-4805-4559-93cb-33e622b3a537" containerName="discovery" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869944 4848 cpu_manager.go:414] "containerMap: removing container" podUID="e2400054-2c0b-4ade-874c-e752a9067b54" containerName="payload-pre-processing" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869951 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2400054-2c0b-4ade-874c-e752a9067b54" containerName="payload-pre-processing" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869984 4848 cpu_manager.go:414] "containerMap: removing container" podUID="29d99320-f1a9-4178-97f6-dc0a088aec0b" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.869993 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d99320-f1a9-4178-97f6-dc0a088aec0b" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870011 4848 cpu_manager.go:414] "containerMap: removing container" podUID="fd7a438e-6705-4f55-8580-fafb7a21f4f0" containerName="maas-api" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870018 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7a438e-6705-4f55-8580-fafb7a21f4f0" containerName="maas-api" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870028 4848 cpu_manager.go:414] "containerMap: removing container" podUID="127e9b91-a76e-46f6-9970-e5dcf8a0ba78" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870035 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="127e9b91-a76e-46f6-9970-e5dcf8a0ba78" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870045 4848 cpu_manager.go:414] "containerMap: removing container" podUID="092690bd-ffd6-4893-96b0-025b55539fd7" containerName="istio-proxy" Oct 30 21:32:46.870035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870051 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="092690bd-ffd6-4893-96b0-025b55539fd7" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870061 4848 cpu_manager.go:414] "containerMap: removing container" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870067 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870078 4848 cpu_manager.go:414] "containerMap: removing container" podUID="72874e47-9bf8-4fcd-98b8-4c51b124244e" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870085 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72874e47-9bf8-4fcd-98b8-4c51b124244e" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870094 4848 cpu_manager.go:414] "containerMap: removing container" podUID="248aa1ff-dd7f-455d-a0db-8881a9af59da" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870101 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="248aa1ff-dd7f-455d-a0db-8881a9af59da" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870126 4848 cpu_manager.go:414] "containerMap: removing container" podUID="69dbc4b2-1701-4de6-a810-795807a04c62" containerName="payload-pre-processing" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870134 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69dbc4b2-1701-4de6-a810-795807a04c62" containerName="payload-pre-processing" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870155 4848 cpu_manager.go:414] "containerMap: removing container" podUID="7f1686ee-2109-4bfa-9964-cc40955a35b9" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870162 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f1686ee-2109-4bfa-9964-cc40955a35b9" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870173 4848 cpu_manager.go:414] "containerMap: removing container" podUID="57e19834-f4ba-495f-a5a8-2b73f8e36cb8" containerName="main" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870180 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57e19834-f4ba-495f-a5a8-2b73f8e36cb8" containerName="main" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870190 4848 cpu_manager.go:414] "containerMap: removing container" podUID="4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870197 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870211 4848 cpu_manager.go:414] "containerMap: removing container" podUID="7f1686ee-2109-4bfa-9964-cc40955a35b9" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870218 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f1686ee-2109-4bfa-9964-cc40955a35b9" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870226 4848 cpu_manager.go:414] "containerMap: removing container" podUID="01a47632-b985-464f-8ee1-4fe60a1b3258" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870233 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a47632-b985-464f-8ee1-4fe60a1b3258" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870241 4848 cpu_manager.go:414] "containerMap: removing container" podUID="127e9b91-a76e-46f6-9970-e5dcf8a0ba78" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870250 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="127e9b91-a76e-46f6-9970-e5dcf8a0ba78" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870283 4848 cpu_manager.go:414] "containerMap: removing container" podUID="7c946890-dd61-47f5-8c84-ef70e09b2da4" containerName="cleanup" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870291 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c946890-dd61-47f5-8c84-ef70e09b2da4" containerName="cleanup" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870301 4848 cpu_manager.go:414] "containerMap: removing container" podUID="ac5c2e61-6970-43c5-9fb4-1f3991a49466" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870309 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac5c2e61-6970-43c5-9fb4-1f3991a49466" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870328 4848 cpu_manager.go:414] "containerMap: removing container" podUID="691eb0e8-a109-4d75-90c5-d7a7bedd9be0" containerName="cleanup" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870337 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="691eb0e8-a109-4d75-90c5-d7a7bedd9be0" containerName="cleanup" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870347 4848 cpu_manager.go:414] "containerMap: removing container" podUID="3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870354 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870364 4848 cpu_manager.go:414] "containerMap: removing container" podUID="5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870371 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870383 4848 cpu_manager.go:414] "containerMap: removing container" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870389 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870400 4848 cpu_manager.go:414] "containerMap: removing container" podUID="813afec3-2ef0-434f-b4bd-08c9c705c6ec" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870408 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="813afec3-2ef0-434f-b4bd-08c9c705c6ec" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870418 4848 cpu_manager.go:414] "containerMap: removing container" podUID="29d99320-f1a9-4178-97f6-dc0a088aec0b" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870425 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29d99320-f1a9-4178-97f6-dc0a088aec0b" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870435 4848 cpu_manager.go:414] "containerMap: removing container" podUID="11a4ae21-6152-44d6-9327-18c9a856261c" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870441 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a4ae21-6152-44d6-9327-18c9a856261c" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870454 4848 cpu_manager.go:414] "containerMap: removing container" podUID="11a4ae21-6152-44d6-9327-18c9a856261c" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870462 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11a4ae21-6152-44d6-9327-18c9a856261c" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870471 4848 cpu_manager.go:414] "containerMap: removing container" podUID="28231432-c0bf-4c3b-879e-40acaceddeec" containerName="cleanup" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870478 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28231432-c0bf-4c3b-879e-40acaceddeec" containerName="cleanup" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870488 4848 cpu_manager.go:414] "containerMap: removing container" podUID="9d3d52ea-ca0a-456e-bf85-896e16fedc84" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870494 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3d52ea-ca0a-456e-bf85-896e16fedc84" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870509 4848 cpu_manager.go:414] "containerMap: removing container" podUID="906f25cf-0aea-484b-9dcc-5302dfb1c5e4" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870516 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="906f25cf-0aea-484b-9dcc-5302dfb1c5e4" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870525 4848 cpu_manager.go:414] "containerMap: removing container" podUID="9e35d342-d32b-4cbb-8249-6a4c12ce82cd" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870533 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e35d342-d32b-4cbb-8249-6a4c12ce82cd" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870655 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd7a438e-6705-4f55-8580-fafb7a21f4f0" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870670 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="127e9b91-a76e-46f6-9970-e5dcf8a0ba78" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870680 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="092690bd-ffd6-4893-96b0-025b55539fd7" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870691 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" containerName="main" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870703 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="72874e47-9bf8-4fcd-98b8-4c51b124244e" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870711 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="248aa1ff-dd7f-455d-a0db-8881a9af59da" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870721 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="69dbc4b2-1701-4de6-a810-795807a04c62" containerName="payload-pre-processing" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870732 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c946890-dd61-47f5-8c84-ef70e09b2da4" containerName="cleanup" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870740 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f1686ee-2109-4bfa-9964-cc40955a35b9" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870749 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="57e19834-f4ba-495f-a5a8-2b73f8e36cb8" containerName="main" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870757 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870783 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="28231432-c0bf-4c3b-879e-40acaceddeec" containerName="cleanup" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870791 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f1686ee-2109-4bfa-9964-cc40955a35b9" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870799 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a47632-b985-464f-8ee1-4fe60a1b3258" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870807 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="127e9b91-a76e-46f6-9970-e5dcf8a0ba78" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870814 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c946890-dd61-47f5-8c84-ef70e09b2da4" containerName="cleanup" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870822 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac5c2e61-6970-43c5-9fb4-1f3991a49466" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870832 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="691eb0e8-a109-4d75-90c5-d7a7bedd9be0" containerName="cleanup" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870840 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="691eb0e8-a109-4d75-90c5-d7a7bedd9be0" containerName="cleanup" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870848 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="3acee8e9-ef3d-412d-8d3e-c5e1ef1d92cb" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870857 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870865 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="9184c819-a720-46b2-992b-65a4087be757" containerName="main" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870873 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="813afec3-2ef0-434f-b4bd-08c9c705c6ec" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870882 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d99320-f1a9-4178-97f6-dc0a088aec0b" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870889 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a4ae21-6152-44d6-9327-18c9a856261c" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870898 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="11a4ae21-6152-44d6-9327-18c9a856261c" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870905 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3d52ea-ca0a-456e-bf85-896e16fedc84" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870913 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="906f25cf-0aea-484b-9dcc-5302dfb1c5e4" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870923 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e35d342-d32b-4cbb-8249-6a4c12ce82cd" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870932 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870940 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="797a8c75-fb91-4d33-8833-e062f5cd01f0" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870950 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee6e41be-d07f-4346-a538-8bd375b21b9b" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870960 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f1a3cb7-8aac-4c2c-8c6c-b2a7f4507c2d" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870968 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a47632-b985-464f-8ee1-4fe60a1b3258" containerName="istio-proxy" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870978 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="3974a64c-bb4d-436e-bcfe-da5277f71f8c" containerName="maas-api" Oct 30 21:32:46.872187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.870985 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebde61df-6d3a-43f2-8ab0-e61c220a5259" containerName="istio-proxy" Oct 30 21:32:46.876341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.873168 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa84afb2-c58a-4ac6-be54-bdfef411fd2b" containerName="payload-pre-processing" Oct 30 21:32:46.876341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.873182 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="28231432-c0bf-4c3b-879e-40acaceddeec" containerName="cleanup" Oct 30 21:32:46.876341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.873191 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c946890-dd61-47f5-8c84-ef70e09b2da4" containerName="cleanup" Oct 30 21:32:46.876341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.873204 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02fca84-f078-4d46-a842-a9b29f63eaaa" containerName="istio-proxy" Oct 30 21:32:46.876341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.873215 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="81767769-741f-4780-93d2-32406b2a6ba6" containerName="maas-api" Oct 30 21:32:46.876341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.873224 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31751bb-34fe-4e1f-8bcf-2f9b457d3379" containerName="istio-proxy" Oct 30 21:32:46.876341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.873233 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f7e1ecd-c614-4687-96e3-631e5716cf83" containerName="maas-api" Oct 30 21:32:46.876341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.873245 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02fca84-f078-4d46-a842-a9b29f63eaaa" containerName="istio-proxy" Oct 30 21:32:46.876341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.873253 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebde61df-6d3a-43f2-8ab0-e61c220a5259" containerName="istio-proxy" Oct 30 21:32:46.876341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.873263 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f478d7b-4805-4559-93cb-33e622b3a537" containerName="discovery" Oct 30 21:32:46.876341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.873272 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2400054-2c0b-4ade-874c-e752a9067b54" containerName="payload-pre-processing" Oct 30 21:32:46.876341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.873280 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="29d99320-f1a9-4178-97f6-dc0a088aec0b" containerName="istio-proxy" Oct 30 21:32:46.883161 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.878684 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f21776fa-f824n" Oct 30 21:32:46.883161 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.882233 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f21776fa-f824n"] Oct 30 21:32:46.931487 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.931459 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69dbc4b2-1701-4de6-a810-795807a04c62" path="/var/lib/kubelet/pods/69dbc4b2-1701-4de6-a810-795807a04c62/volumes" Oct 30 21:32:46.931912 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.931898 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9184c819-a720-46b2-992b-65a4087be757" path="/var/lib/kubelet/pods/9184c819-a720-46b2-992b-65a4087be757/volumes" Oct 30 21:32:46.932399 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.932376 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa47948a-4865-45d9-a687-fc7fbcc3a6a8" path="/var/lib/kubelet/pods/fa47948a-4865-45d9-a687-fc7fbcc3a6a8/volumes" Oct 30 21:32:46.970430 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.969939 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:32:46.970430 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.970077 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\"" Oct 30 21:32:46.973753 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.973726 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a88a7285-af5c-4693-93fb-8792d1f8959c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f21776fa-f824n\" (UID: \"a88a7285-af5c-4693-93fb-8792d1f8959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f21776fa-f824n" Oct 30 21:32:46.973987 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.973950 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a88a7285-af5c-4693-93fb-8792d1f8959c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f21776fa-f824n\" (UID: \"a88a7285-af5c-4693-93fb-8792d1f8959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f21776fa-f824n" Oct 30 21:32:46.974168 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.974146 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6q2x\" (UniqueName: \"kubernetes.io/projected/a88a7285-af5c-4693-93fb-8792d1f8959c-kube-api-access-l6q2x\") pod \"maas-api-revoke-sub-f21776fa-f824n\" (UID: \"a88a7285-af5c-4693-93fb-8792d1f8959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f21776fa-f824n" Oct 30 21:32:46.994598 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.988991 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf"] Oct 30 21:32:46.994598 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:46.989230 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" podUID="e57add2c-5ccf-47eb-8adb-d5852289b111" containerName="istio-proxy" containerID="cri-o://5ffdf4d18fd52c60a0ac175d0f926866aaba4fb54478651b05a380dd5c413d07" gracePeriod=30 Oct 30 21:32:47.075625 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:47.075588 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6q2x\" (UniqueName: \"kubernetes.io/projected/a88a7285-af5c-4693-93fb-8792d1f8959c-kube-api-access-l6q2x\") pod \"maas-api-revoke-sub-f21776fa-f824n\" (UID: \"a88a7285-af5c-4693-93fb-8792d1f8959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f21776fa-f824n" Oct 30 21:32:47.075818 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:47.075654 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a88a7285-af5c-4693-93fb-8792d1f8959c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f21776fa-f824n\" (UID: \"a88a7285-af5c-4693-93fb-8792d1f8959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f21776fa-f824n" Oct 30 21:32:47.075818 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:47.075758 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a88a7285-af5c-4693-93fb-8792d1f8959c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f21776fa-f824n\" (UID: \"a88a7285-af5c-4693-93fb-8792d1f8959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f21776fa-f824n" Oct 30 21:32:47.076447 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:47.076417 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a88a7285-af5c-4693-93fb-8792d1f8959c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f21776fa-f824n\" (UID: \"a88a7285-af5c-4693-93fb-8792d1f8959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f21776fa-f824n" Oct 30 21:32:47.078002 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:47.077978 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a88a7285-af5c-4693-93fb-8792d1f8959c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f21776fa-f824n\" (UID: \"a88a7285-af5c-4693-93fb-8792d1f8959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f21776fa-f824n" Oct 30 21:32:47.087821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:47.087795 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6q2x\" (UniqueName: \"kubernetes.io/projected/a88a7285-af5c-4693-93fb-8792d1f8959c-kube-api-access-l6q2x\") pod \"maas-api-revoke-sub-f21776fa-f824n\" (UID: \"a88a7285-af5c-4693-93fb-8792d1f8959c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f21776fa-f824n" Oct 30 21:32:47.199870 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:47.199788 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f21776fa-f824n" Oct 30 21:32:47.474153 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:47.474126 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f21776fa-f824n"] Oct 30 21:32:47.476385 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:32:47.476343 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88a7285_af5c_4693_93fb_8792d1f8959c.slice/crio-2a962aa6a48cfad75dd89d3c96da5fe3e72597314d56a0844ad228022909ceeb WatchSource:0}: Error finding container 2a962aa6a48cfad75dd89d3c96da5fe3e72597314d56a0844ad228022909ceeb: Status 404 returned error can't find the container with id 2a962aa6a48cfad75dd89d3c96da5fe3e72597314d56a0844ad228022909ceeb Oct 30 21:32:47.590210 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:47.590171 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86" podUID="b7f2555a-89d9-4d58-89d5-f076b60af465" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:32:47.656362 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:47.656325 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f21776fa-f824n" event={"ID":"a88a7285-af5c-4693-93fb-8792d1f8959c","Type":"ContainerStarted","Data":"2a962aa6a48cfad75dd89d3c96da5fe3e72597314d56a0844ad228022909ceeb"} Oct 30 21:32:48.662937 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:48.662841 4848 generic.go:356] "Generic (PLEG): container finished" podID="a88a7285-af5c-4693-93fb-8792d1f8959c" containerID="7c3d896b3413d591a238824d72d6480e475fca3ac125483715fa3dea4e09adea" exitCode=0 Oct 30 21:32:48.663370 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:48.662927 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f21776fa-f824n" event={"ID":"a88a7285-af5c-4693-93fb-8792d1f8959c","Type":"ContainerDied","Data":"7c3d896b3413d591a238824d72d6480e475fca3ac125483715fa3dea4e09adea"} Oct 30 21:32:49.526386 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:49.526351 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr" podUID="6867e706-6187-4821-ae3c-141f443718a5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:32:49.526610 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:49.526355 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d" podUID="3d385f07-6dae-4ca6-9e02-eb587a7d6374" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:32:49.825280 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:49.825248 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f21776fa-f824n" Oct 30 21:32:49.904039 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:49.904007 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a88a7285-af5c-4693-93fb-8792d1f8959c-kube-api-access-l6q2x\" (UniqueName: \"kubernetes.io/projected/a88a7285-af5c-4693-93fb-8792d1f8959c-kube-api-access-l6q2x\") pod \"a88a7285-af5c-4693-93fb-8792d1f8959c\" (UID: \"a88a7285-af5c-4693-93fb-8792d1f8959c\") " Oct 30 21:32:49.904242 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:49.904124 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a88a7285-af5c-4693-93fb-8792d1f8959c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a88a7285-af5c-4693-93fb-8792d1f8959c-maas-api-openshift-ca\") pod \"a88a7285-af5c-4693-93fb-8792d1f8959c\" (UID: \"a88a7285-af5c-4693-93fb-8792d1f8959c\") " Oct 30 21:32:49.904242 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:49.904171 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a88a7285-af5c-4693-93fb-8792d1f8959c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a88a7285-af5c-4693-93fb-8792d1f8959c-maas-api-xks-ca\") pod \"a88a7285-af5c-4693-93fb-8792d1f8959c\" (UID: \"a88a7285-af5c-4693-93fb-8792d1f8959c\") " Oct 30 21:32:49.904501 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:49.904477 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a88a7285-af5c-4693-93fb-8792d1f8959c-maas-api-openshift-ca" pod "a88a7285-af5c-4693-93fb-8792d1f8959c" (UID: "a88a7285-af5c-4693-93fb-8792d1f8959c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:49.906135 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:49.906103 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a88a7285-af5c-4693-93fb-8792d1f8959c-maas-api-xks-ca" pod "a88a7285-af5c-4693-93fb-8792d1f8959c" (UID: "a88a7285-af5c-4693-93fb-8792d1f8959c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:49.906245 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:49.906207 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a88a7285-af5c-4693-93fb-8792d1f8959c-kube-api-access-l6q2x" pod "a88a7285-af5c-4693-93fb-8792d1f8959c" (UID: "a88a7285-af5c-4693-93fb-8792d1f8959c"). InnerVolumeSpecName "kube-api-access-l6q2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:50.005396 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:50.005355 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6q2x\" (UniqueName: \"kubernetes.io/projected/a88a7285-af5c-4693-93fb-8792d1f8959c-kube-api-access-l6q2x\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:50.005396 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:50.005387 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a88a7285-af5c-4693-93fb-8792d1f8959c-maas-api-openshift-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:50.005396 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:50.005401 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a88a7285-af5c-4693-93fb-8792d1f8959c-maas-api-xks-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:50.675783 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:50.675747 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f21776fa-f824n" Oct 30 21:32:50.675957 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:50.675802 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f21776fa-f824n" event={"ID":"a88a7285-af5c-4693-93fb-8792d1f8959c","Type":"ContainerDied","Data":"2a962aa6a48cfad75dd89d3c96da5fe3e72597314d56a0844ad228022909ceeb"} Oct 30 21:32:50.675957 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:50.675859 4848 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a962aa6a48cfad75dd89d3c96da5fe3e72597314d56a0844ad228022909ceeb" Oct 30 21:32:51.189235 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.189195 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" containerID="cri-o://4e3e8b05c29097257261f9cfde29e264074efc7fe8741b83da96be17007db80f" gracePeriod=45 Oct 30 21:32:51.207652 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.207617 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" containerID="cri-o://867cc8fba123252ecc85d54799cf002bdead40cab09a401a42cc72fcd045e204" gracePeriod=45 Oct 30 21:32:51.266004 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.265955 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": dial tcp 10.134.0.62:8000: connect: connection refused" Oct 30 21:32:51.474660 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.474631 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 21:32:51.506602 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.506560 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 21:32:51.523461 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.523426 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-home\" (UniqueName: \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-home\") pod \"dc3af69c-3d4e-4aab-98af-9082f94ace11\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " Oct 30 21:32:51.523647 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.523468 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/130328d3-b4f3-4123-9116-c5e462b0c8dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/130328d3-b4f3-4123-9116-c5e462b0c8dc-tls-certs\") pod \"130328d3-b4f3-4123-9116-c5e462b0c8dc\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " Oct 30 21:32:51.523647 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.523488 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-tmp-dir\") pod \"130328d3-b4f3-4123-9116-c5e462b0c8dc\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " Oct 30 21:32:51.523647 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.523528 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-model-cache\") pod \"130328d3-b4f3-4123-9116-c5e462b0c8dc\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " Oct 30 21:32:51.523647 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.523570 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-dshm\") pod \"dc3af69c-3d4e-4aab-98af-9082f94ace11\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " Oct 30 21:32:51.523647 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.523600 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-tmp-dir\") pod \"dc3af69c-3d4e-4aab-98af-9082f94ace11\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " Oct 30 21:32:51.523963 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.523664 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/130328d3-b4f3-4123-9116-c5e462b0c8dc-kube-api-access-t5h96\" (UniqueName: \"kubernetes.io/projected/130328d3-b4f3-4123-9116-c5e462b0c8dc-kube-api-access-t5h96\") pod \"130328d3-b4f3-4123-9116-c5e462b0c8dc\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " Oct 30 21:32:51.523963 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.523709 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-home\" (UniqueName: \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-home\") pod \"130328d3-b4f3-4123-9116-c5e462b0c8dc\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " Oct 30 21:32:51.523963 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.523746 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-home" pod "dc3af69c-3d4e-4aab-98af-9082f94ace11" (UID: "dc3af69c-3d4e-4aab-98af-9082f94ace11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:51.523963 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.523778 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-dshm\") pod \"130328d3-b4f3-4123-9116-c5e462b0c8dc\" (UID: \"130328d3-b4f3-4123-9116-c5e462b0c8dc\") " Oct 30 21:32:51.523963 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.523812 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-model-cache\") pod \"dc3af69c-3d4e-4aab-98af-9082f94ace11\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " Oct 30 21:32:51.523963 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.523856 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc3af69c-3d4e-4aab-98af-9082f94ace11-kube-api-access-rk4nn\" (UniqueName: \"kubernetes.io/projected/dc3af69c-3d4e-4aab-98af-9082f94ace11-kube-api-access-rk4nn\") pod \"dc3af69c-3d4e-4aab-98af-9082f94ace11\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " Oct 30 21:32:51.523963 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.523917 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc3af69c-3d4e-4aab-98af-9082f94ace11-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3af69c-3d4e-4aab-98af-9082f94ace11-tls-certs\") pod \"dc3af69c-3d4e-4aab-98af-9082f94ace11\" (UID: \"dc3af69c-3d4e-4aab-98af-9082f94ace11\") " Oct 30 21:32:51.524347 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.523993 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-tmp-dir" pod "130328d3-b4f3-4123-9116-c5e462b0c8dc" (UID: "130328d3-b4f3-4123-9116-c5e462b0c8dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:51.524347 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.524327 4848 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-home\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:51.524469 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.524347 4848 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-tmp-dir\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:51.524469 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.524421 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-model-cache" pod "130328d3-b4f3-4123-9116-c5e462b0c8dc" (UID: "130328d3-b4f3-4123-9116-c5e462b0c8dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:51.524469 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.524433 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-model-cache" pod "dc3af69c-3d4e-4aab-98af-9082f94ace11" (UID: "dc3af69c-3d4e-4aab-98af-9082f94ace11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:51.524698 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.524648 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-home" pod "130328d3-b4f3-4123-9116-c5e462b0c8dc" (UID: "130328d3-b4f3-4123-9116-c5e462b0c8dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:51.524808 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.524748 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-tmp-dir" pod "dc3af69c-3d4e-4aab-98af-9082f94ace11" (UID: "dc3af69c-3d4e-4aab-98af-9082f94ace11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:51.529334 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.529017 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc3af69c-3d4e-4aab-98af-9082f94ace11-tls-certs" pod "dc3af69c-3d4e-4aab-98af-9082f94ace11" (UID: "dc3af69c-3d4e-4aab-98af-9082f94ace11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:51.530684 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.530627 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/130328d3-b4f3-4123-9116-c5e462b0c8dc-tls-certs" pod "130328d3-b4f3-4123-9116-c5e462b0c8dc" (UID: "130328d3-b4f3-4123-9116-c5e462b0c8dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:51.531096 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.531072 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130328d3-b4f3-4123-9116-c5e462b0c8dc-kube-api-access-t5h96" pod "130328d3-b4f3-4123-9116-c5e462b0c8dc" (UID: "130328d3-b4f3-4123-9116-c5e462b0c8dc"). InnerVolumeSpecName "kube-api-access-t5h96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:51.531199 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.531151 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-dshm" pod "130328d3-b4f3-4123-9116-c5e462b0c8dc" (UID: "130328d3-b4f3-4123-9116-c5e462b0c8dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:51.531199 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.531181 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3af69c-3d4e-4aab-98af-9082f94ace11-kube-api-access-rk4nn" pod "dc3af69c-3d4e-4aab-98af-9082f94ace11" (UID: "dc3af69c-3d4e-4aab-98af-9082f94ace11"). InnerVolumeSpecName "kube-api-access-rk4nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:51.531455 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.531414 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-dshm" pod "dc3af69c-3d4e-4aab-98af-9082f94ace11" (UID: "dc3af69c-3d4e-4aab-98af-9082f94ace11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:51.625956 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.625918 4848 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-model-cache\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:51.625956 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.625952 4848 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-dshm\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:51.626173 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.625968 4848 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-tmp-dir\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:51.626173 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.625982 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5h96\" (UniqueName: \"kubernetes.io/projected/130328d3-b4f3-4123-9116-c5e462b0c8dc-kube-api-access-t5h96\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:51.626173 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.626000 4848 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-home\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:51.626173 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.626013 4848 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/130328d3-b4f3-4123-9116-c5e462b0c8dc-dshm\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:51.626173 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.626026 4848 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc3af69c-3d4e-4aab-98af-9082f94ace11-model-cache\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:51.626173 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.626040 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk4nn\" (UniqueName: \"kubernetes.io/projected/dc3af69c-3d4e-4aab-98af-9082f94ace11-kube-api-access-rk4nn\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:51.626173 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.626056 4848 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc3af69c-3d4e-4aab-98af-9082f94ace11-tls-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:51.626173 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.626070 4848 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/130328d3-b4f3-4123-9116-c5e462b0c8dc-tls-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:51.684229 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.684196 4848 generic.go:356] "Generic (PLEG): container finished" podID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerID="867cc8fba123252ecc85d54799cf002bdead40cab09a401a42cc72fcd045e204" exitCode=0 Oct 30 21:32:51.684434 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.684243 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" Oct 30 21:32:51.684434 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.684276 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" event={"ID":"dc3af69c-3d4e-4aab-98af-9082f94ace11","Type":"ContainerDied","Data":"867cc8fba123252ecc85d54799cf002bdead40cab09a401a42cc72fcd045e204"} Oct 30 21:32:51.684434 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.684317 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp" event={"ID":"dc3af69c-3d4e-4aab-98af-9082f94ace11","Type":"ContainerDied","Data":"5e98265dd18080f2a490813034f59aea9addc8d17a28618e8c73c0206e0cab2a"} Oct 30 21:32:51.684434 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.684335 4848 scope.go:122] "RemoveContainer" containerID="867cc8fba123252ecc85d54799cf002bdead40cab09a401a42cc72fcd045e204" Oct 30 21:32:51.685868 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.685848 4848 generic.go:356] "Generic (PLEG): container finished" podID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerID="4e3e8b05c29097257261f9cfde29e264074efc7fe8741b83da96be17007db80f" exitCode=0 Oct 30 21:32:51.685964 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.685890 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" event={"ID":"130328d3-b4f3-4123-9116-c5e462b0c8dc","Type":"ContainerDied","Data":"4e3e8b05c29097257261f9cfde29e264074efc7fe8741b83da96be17007db80f"} Oct 30 21:32:51.685964 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.685908 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" Oct 30 21:32:51.685964 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.685924 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4" event={"ID":"130328d3-b4f3-4123-9116-c5e462b0c8dc","Type":"ContainerDied","Data":"9cdd6b55193a665f551ac7d7eb5fad35f09acc4cc00eb9e2ba121bf87a2d5fa8"} Oct 30 21:32:51.694578 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.694559 4848 scope.go:122] "RemoveContainer" containerID="867cc8fba123252ecc85d54799cf002bdead40cab09a401a42cc72fcd045e204" Oct 30 21:32:51.694956 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:32:51.694886 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"867cc8fba123252ecc85d54799cf002bdead40cab09a401a42cc72fcd045e204\": container with ID starting with 867cc8fba123252ecc85d54799cf002bdead40cab09a401a42cc72fcd045e204 not found: ID does not exist" containerID="867cc8fba123252ecc85d54799cf002bdead40cab09a401a42cc72fcd045e204" Oct 30 21:32:51.694956 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.694921 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"867cc8fba123252ecc85d54799cf002bdead40cab09a401a42cc72fcd045e204"} err="failed to get container status \"867cc8fba123252ecc85d54799cf002bdead40cab09a401a42cc72fcd045e204\": rpc error: code = NotFound desc = could not find container \"867cc8fba123252ecc85d54799cf002bdead40cab09a401a42cc72fcd045e204\": container with ID starting with 867cc8fba123252ecc85d54799cf002bdead40cab09a401a42cc72fcd045e204 not found: ID does not exist" Oct 30 21:32:51.694956 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.694945 4848 scope.go:122] "RemoveContainer" containerID="4e3e8b05c29097257261f9cfde29e264074efc7fe8741b83da96be17007db80f" Oct 30 21:32:51.706903 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.706886 4848 scope.go:122] "RemoveContainer" containerID="4e3e8b05c29097257261f9cfde29e264074efc7fe8741b83da96be17007db80f" Oct 30 21:32:51.707229 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:32:51.707204 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3e8b05c29097257261f9cfde29e264074efc7fe8741b83da96be17007db80f\": container with ID starting with 4e3e8b05c29097257261f9cfde29e264074efc7fe8741b83da96be17007db80f not found: ID does not exist" containerID="4e3e8b05c29097257261f9cfde29e264074efc7fe8741b83da96be17007db80f" Oct 30 21:32:51.707284 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.707241 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3e8b05c29097257261f9cfde29e264074efc7fe8741b83da96be17007db80f"} err="failed to get container status \"4e3e8b05c29097257261f9cfde29e264074efc7fe8741b83da96be17007db80f\": rpc error: code = NotFound desc = could not find container \"4e3e8b05c29097257261f9cfde29e264074efc7fe8741b83da96be17007db80f\": container with ID starting with 4e3e8b05c29097257261f9cfde29e264074efc7fe8741b83da96be17007db80f not found: ID does not exist" Oct 30 21:32:51.713446 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.713426 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp"] Oct 30 21:32:51.717375 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.717353 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f9a819/premium-w3-f9a819-kserve-7b56fdfd69-th6hp"] Oct 30 21:32:51.728716 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.728691 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4"] Oct 30 21:32:51.733247 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:51.733224 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-f9a819/distinct-2-w3-f9a819-kserve-86b579fd95-ssmz4"] Oct 30 21:32:52.249720 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.249691 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 21:32:52.332505 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.332409 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-envoy\") pod \"e57add2c-5ccf-47eb-8adb-d5852289b111\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " Oct 30 21:32:52.332505 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.332483 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-workload-certs\") pod \"e57add2c-5ccf-47eb-8adb-d5852289b111\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " Oct 30 21:32:52.332750 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.332517 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e57add2c-5ccf-47eb-8adb-d5852289b111-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57add2c-5ccf-47eb-8adb-d5852289b111-istiod-ca-cert\") pod \"e57add2c-5ccf-47eb-8adb-d5852289b111\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " Oct 30 21:32:52.332750 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.332572 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-podinfo\") pod \"e57add2c-5ccf-47eb-8adb-d5852289b111\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " Oct 30 21:32:52.332750 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.332601 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-token\" (UniqueName: \"kubernetes.io/projected/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-token\") pod \"e57add2c-5ccf-47eb-8adb-d5852289b111\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " Oct 30 21:32:52.332750 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.332634 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e57add2c-5ccf-47eb-8adb-d5852289b111-kube-api-access-2528m\" (UniqueName: \"kubernetes.io/projected/e57add2c-5ccf-47eb-8adb-d5852289b111-kube-api-access-2528m\") pod \"e57add2c-5ccf-47eb-8adb-d5852289b111\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " Oct 30 21:32:52.332750 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.332697 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-workload-socket\") pod \"e57add2c-5ccf-47eb-8adb-d5852289b111\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " Oct 30 21:32:52.333059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.332809 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-credential-socket\") pod \"e57add2c-5ccf-47eb-8adb-d5852289b111\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " Oct 30 21:32:52.333059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.332847 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-data\") pod \"e57add2c-5ccf-47eb-8adb-d5852289b111\" (UID: \"e57add2c-5ccf-47eb-8adb-d5852289b111\") " Oct 30 21:32:52.333059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.332816 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-workload-certs" pod "e57add2c-5ccf-47eb-8adb-d5852289b111" (UID: "e57add2c-5ccf-47eb-8adb-d5852289b111"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:52.333059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.332968 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e57add2c-5ccf-47eb-8adb-d5852289b111-istiod-ca-cert" pod "e57add2c-5ccf-47eb-8adb-d5852289b111" (UID: "e57add2c-5ccf-47eb-8adb-d5852289b111"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:52.333250 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.333085 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-workload-socket" pod "e57add2c-5ccf-47eb-8adb-d5852289b111" (UID: "e57add2c-5ccf-47eb-8adb-d5852289b111"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:52.333250 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.333218 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-credential-socket" pod "e57add2c-5ccf-47eb-8adb-d5852289b111" (UID: "e57add2c-5ccf-47eb-8adb-d5852289b111"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:52.333471 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.333449 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:52.333471 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.333461 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-data" pod "e57add2c-5ccf-47eb-8adb-d5852289b111" (UID: "e57add2c-5ccf-47eb-8adb-d5852289b111"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:52.333619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.333478 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e57add2c-5ccf-47eb-8adb-d5852289b111-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:52.333619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.333494 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:52.333619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.333510 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:52.334995 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.334971 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-envoy" pod "e57add2c-5ccf-47eb-8adb-d5852289b111" (UID: "e57add2c-5ccf-47eb-8adb-d5852289b111"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:52.335101 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.335061 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-token" pod "e57add2c-5ccf-47eb-8adb-d5852289b111" (UID: "e57add2c-5ccf-47eb-8adb-d5852289b111"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:52.335169 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.335154 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57add2c-5ccf-47eb-8adb-d5852289b111-kube-api-access-2528m" pod "e57add2c-5ccf-47eb-8adb-d5852289b111" (UID: "e57add2c-5ccf-47eb-8adb-d5852289b111"). InnerVolumeSpecName "kube-api-access-2528m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:52.335258 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.335244 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-podinfo" pod "e57add2c-5ccf-47eb-8adb-d5852289b111" (UID: "e57add2c-5ccf-47eb-8adb-d5852289b111"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:32:52.434222 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.434188 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:52.434222 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.434217 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:52.434222 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.434228 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2528m\" (UniqueName: \"kubernetes.io/projected/e57add2c-5ccf-47eb-8adb-d5852289b111-kube-api-access-2528m\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:52.434464 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.434238 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:52.434464 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.434248 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e57add2c-5ccf-47eb-8adb-d5852289b111-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:32:52.509740 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:32:52.509702 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:32:52.693783 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.693731 4848 generic.go:356] "Generic (PLEG): container finished" podID="e57add2c-5ccf-47eb-8adb-d5852289b111" containerID="5ffdf4d18fd52c60a0ac175d0f926866aaba4fb54478651b05a380dd5c413d07" exitCode=0 Oct 30 21:32:52.694001 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.693796 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" Oct 30 21:32:52.694001 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.693807 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" event={"ID":"e57add2c-5ccf-47eb-8adb-d5852289b111","Type":"ContainerDied","Data":"5ffdf4d18fd52c60a0ac175d0f926866aaba4fb54478651b05a380dd5c413d07"} Oct 30 21:32:52.694001 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.693887 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf" event={"ID":"e57add2c-5ccf-47eb-8adb-d5852289b111","Type":"ContainerDied","Data":"39403cc954c0004505ab47741e5bfb9f04db9ed09606dd8b0bd0458f278f6039"} Oct 30 21:32:52.694001 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.693911 4848 scope.go:122] "RemoveContainer" containerID="5ffdf4d18fd52c60a0ac175d0f926866aaba4fb54478651b05a380dd5c413d07" Oct 30 21:32:52.706654 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.706635 4848 scope.go:122] "RemoveContainer" containerID="8e74ab0c9e3f60b54bdd7640b98965b377814024591e577b3ef4af4a82d75b86" Oct 30 21:32:52.715397 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.715376 4848 scope.go:122] "RemoveContainer" containerID="5ffdf4d18fd52c60a0ac175d0f926866aaba4fb54478651b05a380dd5c413d07" Oct 30 21:32:52.715658 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:32:52.715639 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ffdf4d18fd52c60a0ac175d0f926866aaba4fb54478651b05a380dd5c413d07\": container with ID starting with 5ffdf4d18fd52c60a0ac175d0f926866aaba4fb54478651b05a380dd5c413d07 not found: ID does not exist" containerID="5ffdf4d18fd52c60a0ac175d0f926866aaba4fb54478651b05a380dd5c413d07" Oct 30 21:32:52.715707 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.715666 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ffdf4d18fd52c60a0ac175d0f926866aaba4fb54478651b05a380dd5c413d07"} err="failed to get container status \"5ffdf4d18fd52c60a0ac175d0f926866aaba4fb54478651b05a380dd5c413d07\": rpc error: code = NotFound desc = could not find container \"5ffdf4d18fd52c60a0ac175d0f926866aaba4fb54478651b05a380dd5c413d07\": container with ID starting with 5ffdf4d18fd52c60a0ac175d0f926866aaba4fb54478651b05a380dd5c413d07 not found: ID does not exist" Oct 30 21:32:52.715707 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.715681 4848 scope.go:122] "RemoveContainer" containerID="8e74ab0c9e3f60b54bdd7640b98965b377814024591e577b3ef4af4a82d75b86" Oct 30 21:32:52.715933 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:32:52.715914 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e74ab0c9e3f60b54bdd7640b98965b377814024591e577b3ef4af4a82d75b86\": container with ID starting with 8e74ab0c9e3f60b54bdd7640b98965b377814024591e577b3ef4af4a82d75b86 not found: ID does not exist" containerID="8e74ab0c9e3f60b54bdd7640b98965b377814024591e577b3ef4af4a82d75b86" Oct 30 21:32:52.715978 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.715940 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e74ab0c9e3f60b54bdd7640b98965b377814024591e577b3ef4af4a82d75b86"} err="failed to get container status \"8e74ab0c9e3f60b54bdd7640b98965b377814024591e577b3ef4af4a82d75b86\": rpc error: code = NotFound desc = could not find container \"8e74ab0c9e3f60b54bdd7640b98965b377814024591e577b3ef4af4a82d75b86\": container with ID starting with 8e74ab0c9e3f60b54bdd7640b98965b377814024591e577b3ef4af4a82d75b86 not found: ID does not exist" Oct 30 21:32:52.718877 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.718855 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf"] Oct 30 21:32:52.723552 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.723528 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-53aa47-openshift-default-9cbf98959-c7dvf"] Oct 30 21:32:52.931228 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.931197 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" path="/var/lib/kubelet/pods/130328d3-b4f3-4123-9116-c5e462b0c8dc/volumes" Oct 30 21:32:52.931626 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.931612 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" path="/var/lib/kubelet/pods/dc3af69c-3d4e-4aab-98af-9082f94ace11/volumes" Oct 30 21:32:52.932040 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:52.932026 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e57add2c-5ccf-47eb-8adb-d5852289b111" path="/var/lib/kubelet/pods/e57add2c-5ccf-47eb-8adb-d5852289b111/volumes" Oct 30 21:32:57.591574 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:57.591529 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86" podUID="b7f2555a-89d9-4d58-89d5-f076b60af465" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:32:57.831241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:57.831209 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq"] Oct 30 21:32:57.831448 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:57.831421 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" podUID="a1bfc658-f4c9-4280-91e9-5081b0f9416e" containerName="istio-proxy" containerID="cri-o://2334048ffb6810084c8a07d391b456f6a03f946b15926efabd11aafdde563eea" gracePeriod=30 Oct 30 21:32:59.525969 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:59.525923 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr" podUID="6867e706-6187-4821-ae3c-141f443718a5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:32:59.526400 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:32:59.526018 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d" podUID="3d385f07-6dae-4ca6-9e02-eb587a7d6374" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:33:00.619124 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:00.619091 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw"] Oct 30 21:33:00.619617 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:00.619337 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" podUID="a3aee534-70d3-4567-8a70-063a4e684526" containerName="maas-api" containerID="cri-o://380ee59b786cee0efee72b3fa65f1e8e0f87218a52cc80f9ca40456314c1f884" gracePeriod=30 Oct 30 21:33:00.741977 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:00.741942 4848 generic.go:356] "Generic (PLEG): container finished" podID="a3aee534-70d3-4567-8a70-063a4e684526" containerID="380ee59b786cee0efee72b3fa65f1e8e0f87218a52cc80f9ca40456314c1f884" exitCode=0 Oct 30 21:33:00.742110 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:00.741978 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" event={"ID":"a3aee534-70d3-4567-8a70-063a4e684526","Type":"ContainerDied","Data":"380ee59b786cee0efee72b3fa65f1e8e0f87218a52cc80f9ca40456314c1f884"} Oct 30 21:33:00.925139 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:00.925088 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" Oct 30 21:33:01.025722 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.025692 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3aee534-70d3-4567-8a70-063a4e684526-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3aee534-70d3-4567-8a70-063a4e684526-openshift-service-ca\") pod \"a3aee534-70d3-4567-8a70-063a4e684526\" (UID: \"a3aee534-70d3-4567-8a70-063a4e684526\") " Oct 30 21:33:01.025904 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.025785 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3aee534-70d3-4567-8a70-063a4e684526-kube-api-access-x7t9z\" (UniqueName: \"kubernetes.io/projected/a3aee534-70d3-4567-8a70-063a4e684526-kube-api-access-x7t9z\") pod \"a3aee534-70d3-4567-8a70-063a4e684526\" (UID: \"a3aee534-70d3-4567-8a70-063a4e684526\") " Oct 30 21:33:01.025904 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.025836 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3aee534-70d3-4567-8a70-063a4e684526-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3aee534-70d3-4567-8a70-063a4e684526-maas-api-tls\") pod \"a3aee534-70d3-4567-8a70-063a4e684526\" (UID: \"a3aee534-70d3-4567-8a70-063a4e684526\") " Oct 30 21:33:01.025904 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.025887 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3aee534-70d3-4567-8a70-063a4e684526-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3aee534-70d3-4567-8a70-063a4e684526-metrics-certs\") pod \"a3aee534-70d3-4567-8a70-063a4e684526\" (UID: \"a3aee534-70d3-4567-8a70-063a4e684526\") " Oct 30 21:33:01.026105 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.026063 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3aee534-70d3-4567-8a70-063a4e684526-openshift-service-ca" pod "a3aee534-70d3-4567-8a70-063a4e684526" (UID: "a3aee534-70d3-4567-8a70-063a4e684526"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:01.026344 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.026316 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3aee534-70d3-4567-8a70-063a4e684526-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:01.028157 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.028122 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3aee534-70d3-4567-8a70-063a4e684526-maas-api-tls" pod "a3aee534-70d3-4567-8a70-063a4e684526" (UID: "a3aee534-70d3-4567-8a70-063a4e684526"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:01.028408 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.028378 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3aee534-70d3-4567-8a70-063a4e684526-metrics-certs" pod "a3aee534-70d3-4567-8a70-063a4e684526" (UID: "a3aee534-70d3-4567-8a70-063a4e684526"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:01.028509 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.028446 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3aee534-70d3-4567-8a70-063a4e684526-kube-api-access-x7t9z" pod "a3aee534-70d3-4567-8a70-063a4e684526" (UID: "a3aee534-70d3-4567-8a70-063a4e684526"). InnerVolumeSpecName "kube-api-access-x7t9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:01.028557 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.028527 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86" Oct 30 21:33:01.132854 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.127792 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f2555a-89d9-4d58-89d5-f076b60af465-kube-api-access-w2txq\" (UniqueName: \"kubernetes.io/projected/b7f2555a-89d9-4d58-89d5-f076b60af465-kube-api-access-w2txq\") pod \"b7f2555a-89d9-4d58-89d5-f076b60af465\" (UID: \"b7f2555a-89d9-4d58-89d5-f076b60af465\") " Oct 30 21:33:01.132854 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.127999 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7f2555a-89d9-4d58-89d5-f076b60af465-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f2555a-89d9-4d58-89d5-f076b60af465-plugins-config-volume\") pod \"b7f2555a-89d9-4d58-89d5-f076b60af465\" (UID: \"b7f2555a-89d9-4d58-89d5-f076b60af465\") " Oct 30 21:33:01.132854 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.128283 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3aee534-70d3-4567-8a70-063a4e684526-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:01.132854 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.128305 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7t9z\" (UniqueName: \"kubernetes.io/projected/a3aee534-70d3-4567-8a70-063a4e684526-kube-api-access-x7t9z\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:01.132854 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.128320 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3aee534-70d3-4567-8a70-063a4e684526-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:01.132854 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.128434 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7f2555a-89d9-4d58-89d5-f076b60af465-plugins-config-volume" pod "b7f2555a-89d9-4d58-89d5-f076b60af465" (UID: "b7f2555a-89d9-4d58-89d5-f076b60af465"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:01.132854 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.130577 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f2555a-89d9-4d58-89d5-f076b60af465-kube-api-access-w2txq" pod "b7f2555a-89d9-4d58-89d5-f076b60af465" (UID: "b7f2555a-89d9-4d58-89d5-f076b60af465"). InnerVolumeSpecName "kube-api-access-w2txq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:01.229221 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.229182 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7f2555a-89d9-4d58-89d5-f076b60af465-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:01.229221 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.229217 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2txq\" (UniqueName: \"kubernetes.io/projected/b7f2555a-89d9-4d58-89d5-f076b60af465-kube-api-access-w2txq\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:01.748849 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.748811 4848 generic.go:356] "Generic (PLEG): container finished" podID="b7f2555a-89d9-4d58-89d5-f076b60af465" containerID="272be5d67b61f9b848032198b237badf12d5794f07a997f8245a6ddc98fcdac5" exitCode=0 Oct 30 21:33:01.749315 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.748865 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86" Oct 30 21:33:01.749315 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.748901 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86" event={"ID":"b7f2555a-89d9-4d58-89d5-f076b60af465","Type":"ContainerDied","Data":"272be5d67b61f9b848032198b237badf12d5794f07a997f8245a6ddc98fcdac5"} Oct 30 21:33:01.749315 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.748955 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86" event={"ID":"b7f2555a-89d9-4d58-89d5-f076b60af465","Type":"ContainerDied","Data":"8b65909440080e24039ad39336b098828710f5585e5b304506568a8d143fe49a"} Oct 30 21:33:01.749315 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.748983 4848 scope.go:122] "RemoveContainer" containerID="272be5d67b61f9b848032198b237badf12d5794f07a997f8245a6ddc98fcdac5" Oct 30 21:33:01.750844 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.750823 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" event={"ID":"a3aee534-70d3-4567-8a70-063a4e684526","Type":"ContainerDied","Data":"41245cc0a7eee2a8d7e14f53d22c8d0bbc08a0a366dd6b41d2c9da74ca7165e0"} Oct 30 21:33:01.750948 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.750877 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw" Oct 30 21:33:01.759744 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.759727 4848 scope.go:122] "RemoveContainer" containerID="272be5d67b61f9b848032198b237badf12d5794f07a997f8245a6ddc98fcdac5" Oct 30 21:33:01.760046 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:33:01.760023 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272be5d67b61f9b848032198b237badf12d5794f07a997f8245a6ddc98fcdac5\": container with ID starting with 272be5d67b61f9b848032198b237badf12d5794f07a997f8245a6ddc98fcdac5 not found: ID does not exist" containerID="272be5d67b61f9b848032198b237badf12d5794f07a997f8245a6ddc98fcdac5" Oct 30 21:33:01.760115 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.760058 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272be5d67b61f9b848032198b237badf12d5794f07a997f8245a6ddc98fcdac5"} err="failed to get container status \"272be5d67b61f9b848032198b237badf12d5794f07a997f8245a6ddc98fcdac5\": rpc error: code = NotFound desc = could not find container \"272be5d67b61f9b848032198b237badf12d5794f07a997f8245a6ddc98fcdac5\": container with ID starting with 272be5d67b61f9b848032198b237badf12d5794f07a997f8245a6ddc98fcdac5 not found: ID does not exist" Oct 30 21:33:01.760115 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.760083 4848 scope.go:122] "RemoveContainer" containerID="380ee59b786cee0efee72b3fa65f1e8e0f87218a52cc80f9ca40456314c1f884" Oct 30 21:33:01.775185 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.775162 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86"] Oct 30 21:33:01.780818 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.780796 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-bc102e-779576d99-xvt86"] Oct 30 21:33:01.791245 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.791223 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw"] Oct 30 21:33:01.795492 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:01.795468 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-f9a819-57c675c4c8-8fnvw"] Oct 30 21:33:02.930511 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:02.930471 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3aee534-70d3-4567-8a70-063a4e684526" path="/var/lib/kubelet/pods/a3aee534-70d3-4567-8a70-063a4e684526/volumes" Oct 30 21:33:02.931069 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:02.931049 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7f2555a-89d9-4d58-89d5-f076b60af465" path="/var/lib/kubelet/pods/b7f2555a-89d9-4d58-89d5-f076b60af465/volumes" Oct 30 21:33:03.089150 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.089127 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:33:03.148936 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.148853 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1bfc658-f4c9-4280-91e9-5081b0f9416e-kube-api-access-mjvxp\" (UniqueName: \"kubernetes.io/projected/a1bfc658-f4c9-4280-91e9-5081b0f9416e-kube-api-access-mjvxp\") pod \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " Oct 30 21:33:03.148936 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.148933 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-token\" (UniqueName: \"kubernetes.io/projected/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-token\") pod \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " Oct 30 21:33:03.149175 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.149004 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-podinfo\") pod \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " Oct 30 21:33:03.149175 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.149029 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istiod-ca-cert\") pod \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " Oct 30 21:33:03.149175 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.149073 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-credential-socket\") pod \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " Oct 30 21:33:03.149175 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.149110 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-envoy\") pod \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " Oct 30 21:33:03.149175 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.149141 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-workload-socket\") pod \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " Oct 30 21:33:03.149461 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.149186 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-workload-certs\") pod \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " Oct 30 21:33:03.149461 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.149225 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-data\") pod \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\" (UID: \"a1bfc658-f4c9-4280-91e9-5081b0f9416e\") " Oct 30 21:33:03.149719 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.149536 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istiod-ca-cert" pod "a1bfc658-f4c9-4280-91e9-5081b0f9416e" (UID: "a1bfc658-f4c9-4280-91e9-5081b0f9416e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:03.149719 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.149648 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-workload-socket" pod "a1bfc658-f4c9-4280-91e9-5081b0f9416e" (UID: "a1bfc658-f4c9-4280-91e9-5081b0f9416e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:03.149719 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.149660 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-workload-certs" pod "a1bfc658-f4c9-4280-91e9-5081b0f9416e" (UID: "a1bfc658-f4c9-4280-91e9-5081b0f9416e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:03.149719 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.149685 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-credential-socket" pod "a1bfc658-f4c9-4280-91e9-5081b0f9416e" (UID: "a1bfc658-f4c9-4280-91e9-5081b0f9416e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:03.150064 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.149810 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-data" pod "a1bfc658-f4c9-4280-91e9-5081b0f9416e" (UID: "a1bfc658-f4c9-4280-91e9-5081b0f9416e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:03.151237 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.151212 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-podinfo" pod "a1bfc658-f4c9-4280-91e9-5081b0f9416e" (UID: "a1bfc658-f4c9-4280-91e9-5081b0f9416e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:33:03.151348 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.151288 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bfc658-f4c9-4280-91e9-5081b0f9416e-kube-api-access-mjvxp" pod "a1bfc658-f4c9-4280-91e9-5081b0f9416e" (UID: "a1bfc658-f4c9-4280-91e9-5081b0f9416e"). InnerVolumeSpecName "kube-api-access-mjvxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:03.151410 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.151343 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-envoy" pod "a1bfc658-f4c9-4280-91e9-5081b0f9416e" (UID: "a1bfc658-f4c9-4280-91e9-5081b0f9416e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:03.151468 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.151453 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-token" pod "a1bfc658-f4c9-4280-91e9-5081b0f9416e" (UID: "a1bfc658-f4c9-4280-91e9-5081b0f9416e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:03.250993 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.250961 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:03.250993 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.250988 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:03.251199 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.251002 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:03.251199 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.251015 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:03.251199 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.251028 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:03.251199 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.251043 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:03.251199 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.251054 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:03.251199 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.251067 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjvxp\" (UniqueName: \"kubernetes.io/projected/a1bfc658-f4c9-4280-91e9-5081b0f9416e-kube-api-access-mjvxp\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:03.251199 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.251082 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1bfc658-f4c9-4280-91e9-5081b0f9416e-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:03.765474 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.765442 4848 generic.go:356] "Generic (PLEG): container finished" podID="a1bfc658-f4c9-4280-91e9-5081b0f9416e" containerID="2334048ffb6810084c8a07d391b456f6a03f946b15926efabd11aafdde563eea" exitCode=0 Oct 30 21:33:03.765696 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.765490 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" Oct 30 21:33:03.765696 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.765531 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" event={"ID":"a1bfc658-f4c9-4280-91e9-5081b0f9416e","Type":"ContainerDied","Data":"2334048ffb6810084c8a07d391b456f6a03f946b15926efabd11aafdde563eea"} Oct 30 21:33:03.765696 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.765596 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq" event={"ID":"a1bfc658-f4c9-4280-91e9-5081b0f9416e","Type":"ContainerDied","Data":"7148562e48da5cbdd75459f0dfbf572fecdf3bc064b46b11f0c6178da829b6a1"} Oct 30 21:33:03.765696 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.765629 4848 scope.go:122] "RemoveContainer" containerID="2334048ffb6810084c8a07d391b456f6a03f946b15926efabd11aafdde563eea" Oct 30 21:33:03.780476 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.780458 4848 scope.go:122] "RemoveContainer" containerID="f2f691e82ac81003b1527be23474f45403729a2b5c5d1e4512e21e8d22b4f51e" Oct 30 21:33:03.789589 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.789568 4848 scope.go:122] "RemoveContainer" containerID="2334048ffb6810084c8a07d391b456f6a03f946b15926efabd11aafdde563eea" Oct 30 21:33:03.789916 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:33:03.789887 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2334048ffb6810084c8a07d391b456f6a03f946b15926efabd11aafdde563eea\": container with ID starting with 2334048ffb6810084c8a07d391b456f6a03f946b15926efabd11aafdde563eea not found: ID does not exist" containerID="2334048ffb6810084c8a07d391b456f6a03f946b15926efabd11aafdde563eea" Oct 30 21:33:03.790055 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.789925 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2334048ffb6810084c8a07d391b456f6a03f946b15926efabd11aafdde563eea"} err="failed to get container status \"2334048ffb6810084c8a07d391b456f6a03f946b15926efabd11aafdde563eea\": rpc error: code = NotFound desc = could not find container \"2334048ffb6810084c8a07d391b456f6a03f946b15926efabd11aafdde563eea\": container with ID starting with 2334048ffb6810084c8a07d391b456f6a03f946b15926efabd11aafdde563eea not found: ID does not exist" Oct 30 21:33:03.790055 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.789948 4848 scope.go:122] "RemoveContainer" containerID="f2f691e82ac81003b1527be23474f45403729a2b5c5d1e4512e21e8d22b4f51e" Oct 30 21:33:03.790247 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:33:03.790227 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f691e82ac81003b1527be23474f45403729a2b5c5d1e4512e21e8d22b4f51e\": container with ID starting with f2f691e82ac81003b1527be23474f45403729a2b5c5d1e4512e21e8d22b4f51e not found: ID does not exist" containerID="f2f691e82ac81003b1527be23474f45403729a2b5c5d1e4512e21e8d22b4f51e" Oct 30 21:33:03.790309 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.790256 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f691e82ac81003b1527be23474f45403729a2b5c5d1e4512e21e8d22b4f51e"} err="failed to get container status \"f2f691e82ac81003b1527be23474f45403729a2b5c5d1e4512e21e8d22b4f51e\": rpc error: code = NotFound desc = could not find container \"f2f691e82ac81003b1527be23474f45403729a2b5c5d1e4512e21e8d22b4f51e\": container with ID starting with f2f691e82ac81003b1527be23474f45403729a2b5c5d1e4512e21e8d22b4f51e not found: ID does not exist" Oct 30 21:33:03.801518 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.801498 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq"] Oct 30 21:33:03.807979 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:03.807960 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-6741b1-openshift-default-5954d7485c-w6zrq"] Oct 30 21:33:04.931326 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:04.931296 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1bfc658-f4c9-4280-91e9-5081b0f9416e" path="/var/lib/kubelet/pods/a1bfc658-f4c9-4280-91e9-5081b0f9416e/volumes" Oct 30 21:33:06.743682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.743619 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d" Oct 30 21:33:06.768816 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.768791 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr" Oct 30 21:33:06.782179 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.782154 4848 generic.go:356] "Generic (PLEG): container finished" podID="6867e706-6187-4821-ae3c-141f443718a5" containerID="ba0c27980a493c24c11f878bce0f8fde72b2cb67b173c248d6b2b8377f93d3cd" exitCode=0 Oct 30 21:33:06.782332 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.782209 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr" Oct 30 21:33:06.782332 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.782243 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr" event={"ID":"6867e706-6187-4821-ae3c-141f443718a5","Type":"ContainerDied","Data":"ba0c27980a493c24c11f878bce0f8fde72b2cb67b173c248d6b2b8377f93d3cd"} Oct 30 21:33:06.782332 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.782291 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr" event={"ID":"6867e706-6187-4821-ae3c-141f443718a5","Type":"ContainerDied","Data":"2d8478165477b897ce1ce25f429c8c022c09f5e9e372200f31d57a1db974ee67"} Oct 30 21:33:06.782332 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.782314 4848 scope.go:122] "RemoveContainer" containerID="ba0c27980a493c24c11f878bce0f8fde72b2cb67b173c248d6b2b8377f93d3cd" Oct 30 21:33:06.783660 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.783633 4848 generic.go:356] "Generic (PLEG): container finished" podID="3d385f07-6dae-4ca6-9e02-eb587a7d6374" containerID="53571241600ecbf7f3e5f0e47a2a7f5173a59897a2988b637da5af2f2d94e164" exitCode=0 Oct 30 21:33:06.783791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.783671 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d" Oct 30 21:33:06.783791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.783672 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d" event={"ID":"3d385f07-6dae-4ca6-9e02-eb587a7d6374","Type":"ContainerDied","Data":"53571241600ecbf7f3e5f0e47a2a7f5173a59897a2988b637da5af2f2d94e164"} Oct 30 21:33:06.783791 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.783744 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d" event={"ID":"3d385f07-6dae-4ca6-9e02-eb587a7d6374","Type":"ContainerDied","Data":"b6ef1f234a7febcf833c130e1081bca0076f21d5016db96e7e4bda61d093b8fc"} Oct 30 21:33:06.796522 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.795444 4848 scope.go:122] "RemoveContainer" containerID="ba0c27980a493c24c11f878bce0f8fde72b2cb67b173c248d6b2b8377f93d3cd" Oct 30 21:33:06.796522 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:33:06.795720 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba0c27980a493c24c11f878bce0f8fde72b2cb67b173c248d6b2b8377f93d3cd\": container with ID starting with ba0c27980a493c24c11f878bce0f8fde72b2cb67b173c248d6b2b8377f93d3cd not found: ID does not exist" containerID="ba0c27980a493c24c11f878bce0f8fde72b2cb67b173c248d6b2b8377f93d3cd" Oct 30 21:33:06.796522 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.795742 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0c27980a493c24c11f878bce0f8fde72b2cb67b173c248d6b2b8377f93d3cd"} err="failed to get container status \"ba0c27980a493c24c11f878bce0f8fde72b2cb67b173c248d6b2b8377f93d3cd\": rpc error: code = NotFound desc = could not find container \"ba0c27980a493c24c11f878bce0f8fde72b2cb67b173c248d6b2b8377f93d3cd\": container with ID starting with ba0c27980a493c24c11f878bce0f8fde72b2cb67b173c248d6b2b8377f93d3cd not found: ID does not exist" Oct 30 21:33:06.796522 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.795760 4848 scope.go:122] "RemoveContainer" containerID="53571241600ecbf7f3e5f0e47a2a7f5173a59897a2988b637da5af2f2d94e164" Oct 30 21:33:06.805355 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.805337 4848 scope.go:122] "RemoveContainer" containerID="53571241600ecbf7f3e5f0e47a2a7f5173a59897a2988b637da5af2f2d94e164" Oct 30 21:33:06.805667 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:33:06.805636 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53571241600ecbf7f3e5f0e47a2a7f5173a59897a2988b637da5af2f2d94e164\": container with ID starting with 53571241600ecbf7f3e5f0e47a2a7f5173a59897a2988b637da5af2f2d94e164 not found: ID does not exist" containerID="53571241600ecbf7f3e5f0e47a2a7f5173a59897a2988b637da5af2f2d94e164" Oct 30 21:33:06.805759 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.805663 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53571241600ecbf7f3e5f0e47a2a7f5173a59897a2988b637da5af2f2d94e164"} err="failed to get container status \"53571241600ecbf7f3e5f0e47a2a7f5173a59897a2988b637da5af2f2d94e164\": rpc error: code = NotFound desc = could not find container \"53571241600ecbf7f3e5f0e47a2a7f5173a59897a2988b637da5af2f2d94e164\": container with ID starting with 53571241600ecbf7f3e5f0e47a2a7f5173a59897a2988b637da5af2f2d94e164 not found: ID does not exist" Oct 30 21:33:06.890947 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.890915 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d385f07-6dae-4ca6-9e02-eb587a7d6374-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d385f07-6dae-4ca6-9e02-eb587a7d6374-plugins-config-volume\") pod \"3d385f07-6dae-4ca6-9e02-eb587a7d6374\" (UID: \"3d385f07-6dae-4ca6-9e02-eb587a7d6374\") " Oct 30 21:33:06.891153 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.890979 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d385f07-6dae-4ca6-9e02-eb587a7d6374-kube-api-access-mbk5j\" (UniqueName: \"kubernetes.io/projected/3d385f07-6dae-4ca6-9e02-eb587a7d6374-kube-api-access-mbk5j\") pod \"3d385f07-6dae-4ca6-9e02-eb587a7d6374\" (UID: \"3d385f07-6dae-4ca6-9e02-eb587a7d6374\") " Oct 30 21:33:06.891153 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.890999 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6867e706-6187-4821-ae3c-141f443718a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6867e706-6187-4821-ae3c-141f443718a5-plugins-config-volume\") pod \"6867e706-6187-4821-ae3c-141f443718a5\" (UID: \"6867e706-6187-4821-ae3c-141f443718a5\") " Oct 30 21:33:06.891153 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.891083 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6867e706-6187-4821-ae3c-141f443718a5-kube-api-access-s8sbc\" (UniqueName: \"kubernetes.io/projected/6867e706-6187-4821-ae3c-141f443718a5-kube-api-access-s8sbc\") pod \"6867e706-6187-4821-ae3c-141f443718a5\" (UID: \"6867e706-6187-4821-ae3c-141f443718a5\") " Oct 30 21:33:06.891392 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.891370 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d385f07-6dae-4ca6-9e02-eb587a7d6374-plugins-config-volume" pod "3d385f07-6dae-4ca6-9e02-eb587a7d6374" (UID: "3d385f07-6dae-4ca6-9e02-eb587a7d6374"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:06.891473 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.891454 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6867e706-6187-4821-ae3c-141f443718a5-plugins-config-volume" pod "6867e706-6187-4821-ae3c-141f443718a5" (UID: "6867e706-6187-4821-ae3c-141f443718a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:06.893238 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.893219 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d385f07-6dae-4ca6-9e02-eb587a7d6374-kube-api-access-mbk5j" pod "3d385f07-6dae-4ca6-9e02-eb587a7d6374" (UID: "3d385f07-6dae-4ca6-9e02-eb587a7d6374"). InnerVolumeSpecName "kube-api-access-mbk5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:06.893321 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.893257 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6867e706-6187-4821-ae3c-141f443718a5-kube-api-access-s8sbc" pod "6867e706-6187-4821-ae3c-141f443718a5" (UID: "6867e706-6187-4821-ae3c-141f443718a5"). InnerVolumeSpecName "kube-api-access-s8sbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:06.992154 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.992122 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8sbc\" (UniqueName: \"kubernetes.io/projected/6867e706-6187-4821-ae3c-141f443718a5-kube-api-access-s8sbc\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:06.992154 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.992148 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d385f07-6dae-4ca6-9e02-eb587a7d6374-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:06.992154 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.992158 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbk5j\" (UniqueName: \"kubernetes.io/projected/3d385f07-6dae-4ca6-9e02-eb587a7d6374-kube-api-access-mbk5j\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:06.992370 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:06.992167 4848 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6867e706-6187-4821-ae3c-141f443718a5-plugins-config-volume\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:07.100820 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:07.100744 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d"] Oct 30 21:33:07.107487 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:07.107457 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-f9a819-79dd6b768f-xbw6d"] Oct 30 21:33:07.118808 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:07.118779 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr"] Oct 30 21:33:07.123702 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:07.123678 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-f9a819-5c88dbbb55-bnshr"] Oct 30 21:33:08.931086 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:08.931050 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d385f07-6dae-4ca6-9e02-eb587a7d6374" path="/var/lib/kubelet/pods/3d385f07-6dae-4ca6-9e02-eb587a7d6374/volumes" Oct 30 21:33:08.931632 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:08.931608 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6867e706-6187-4821-ae3c-141f443718a5" path="/var/lib/kubelet/pods/6867e706-6187-4821-ae3c-141f443718a5/volumes" Oct 30 21:33:16.602035 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:16.602004 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 21:33:16.604971 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:16.604943 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-kh5tg_313047e7-bc02-42eb-933b-ab52575af3f7/csi-node-driver-registrar/0.log" Oct 30 21:33:18.202528 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:18.202494 4848 scope.go:122] "RemoveContainer" containerID="2257b757aa8b729d0f2132a0da342937e3b711820c1b9b2a7644e8dd32f9b92e" Oct 30 21:33:18.363111 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:18.363079 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs"] Oct 30 21:33:18.363378 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:18.363330 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" podUID="5c705c9c-fef2-422b-b539-4007e28fedee" containerName="istio-proxy" containerID="cri-o://99739c29e97d34cb7dae40b0820ebf495474db9637a5116103479f887398b8ce" gracePeriod=30 Oct 30 21:33:22.509792 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:33:22.509731 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:33:23.621227 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.621202 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:33:23.661586 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.661512 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c705c9c-fef2-422b-b539-4007e28fedee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c705c9c-fef2-422b-b539-4007e28fedee-istiod-ca-cert\") pod \"5c705c9c-fef2-422b-b539-4007e28fedee\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " Oct 30 21:33:23.661586 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.661563 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c705c9c-fef2-422b-b539-4007e28fedee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c705c9c-fef2-422b-b539-4007e28fedee-istio-podinfo\") pod \"5c705c9c-fef2-422b-b539-4007e28fedee\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " Oct 30 21:33:23.661839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.661594 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-workload-certs\") pod \"5c705c9c-fef2-422b-b539-4007e28fedee\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " Oct 30 21:33:23.661839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.661650 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c705c9c-fef2-422b-b539-4007e28fedee-kube-api-access-9xj9q\" (UniqueName: \"kubernetes.io/projected/5c705c9c-fef2-422b-b539-4007e28fedee-kube-api-access-9xj9q\") pod \"5c705c9c-fef2-422b-b539-4007e28fedee\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " Oct 30 21:33:23.661839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.661754 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c705c9c-fef2-422b-b539-4007e28fedee-istio-token\" (UniqueName: \"kubernetes.io/projected/5c705c9c-fef2-422b-b539-4007e28fedee-istio-token\") pod \"5c705c9c-fef2-422b-b539-4007e28fedee\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " Oct 30 21:33:23.661839 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.661802 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-credential-socket\") pod \"5c705c9c-fef2-422b-b539-4007e28fedee\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " Oct 30 21:33:23.662077 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.661838 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-istio-data\") pod \"5c705c9c-fef2-422b-b539-4007e28fedee\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " Oct 30 21:33:23.662077 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.661901 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-workload-socket\") pod \"5c705c9c-fef2-422b-b539-4007e28fedee\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " Oct 30 21:33:23.662077 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.661931 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-istio-envoy\") pod \"5c705c9c-fef2-422b-b539-4007e28fedee\" (UID: \"5c705c9c-fef2-422b-b539-4007e28fedee\") " Oct 30 21:33:23.662077 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.661953 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c705c9c-fef2-422b-b539-4007e28fedee-istiod-ca-cert" pod "5c705c9c-fef2-422b-b539-4007e28fedee" (UID: "5c705c9c-fef2-422b-b539-4007e28fedee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:23.662077 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.662012 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-workload-certs" pod "5c705c9c-fef2-422b-b539-4007e28fedee" (UID: "5c705c9c-fef2-422b-b539-4007e28fedee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:23.662369 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.662184 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-credential-socket" pod "5c705c9c-fef2-422b-b539-4007e28fedee" (UID: "5c705c9c-fef2-422b-b539-4007e28fedee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:23.662369 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.662251 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-istio-data" pod "5c705c9c-fef2-422b-b539-4007e28fedee" (UID: "5c705c9c-fef2-422b-b539-4007e28fedee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:23.662488 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.662394 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-workload-socket" pod "5c705c9c-fef2-422b-b539-4007e28fedee" (UID: "5c705c9c-fef2-422b-b539-4007e28fedee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:23.662488 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.662441 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c705c9c-fef2-422b-b539-4007e28fedee-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:23.662488 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.662462 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:23.662713 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.662492 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:23.662713 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.662508 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:23.664005 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.663982 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-istio-envoy" pod "5c705c9c-fef2-422b-b539-4007e28fedee" (UID: "5c705c9c-fef2-422b-b539-4007e28fedee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:23.664117 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.664026 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c705c9c-fef2-422b-b539-4007e28fedee-kube-api-access-9xj9q" pod "5c705c9c-fef2-422b-b539-4007e28fedee" (UID: "5c705c9c-fef2-422b-b539-4007e28fedee"). InnerVolumeSpecName "kube-api-access-9xj9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:23.664159 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.664112 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c705c9c-fef2-422b-b539-4007e28fedee-istio-podinfo" pod "5c705c9c-fef2-422b-b539-4007e28fedee" (UID: "5c705c9c-fef2-422b-b539-4007e28fedee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:33:23.664590 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.664572 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c705c9c-fef2-422b-b539-4007e28fedee-istio-token" pod "5c705c9c-fef2-422b-b539-4007e28fedee" (UID: "5c705c9c-fef2-422b-b539-4007e28fedee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:23.763957 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.763925 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:23.763957 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.763953 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c705c9c-fef2-422b-b539-4007e28fedee-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:23.763957 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.763963 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xj9q\" (UniqueName: \"kubernetes.io/projected/5c705c9c-fef2-422b-b539-4007e28fedee-kube-api-access-9xj9q\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:23.764204 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.763972 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c705c9c-fef2-422b-b539-4007e28fedee-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:23.764204 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.763981 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c705c9c-fef2-422b-b539-4007e28fedee-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:23.889682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.889639 4848 generic.go:356] "Generic (PLEG): container finished" podID="5c705c9c-fef2-422b-b539-4007e28fedee" containerID="99739c29e97d34cb7dae40b0820ebf495474db9637a5116103479f887398b8ce" exitCode=0 Oct 30 21:33:23.889909 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.889693 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" Oct 30 21:33:23.889909 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.889715 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" event={"ID":"5c705c9c-fef2-422b-b539-4007e28fedee","Type":"ContainerDied","Data":"99739c29e97d34cb7dae40b0820ebf495474db9637a5116103479f887398b8ce"} Oct 30 21:33:23.889909 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.889787 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs" event={"ID":"5c705c9c-fef2-422b-b539-4007e28fedee","Type":"ContainerDied","Data":"3b0b00ed351e7c6a8e86444dd27c5920270a3dcb68447c1b6b753a98d80a7396"} Oct 30 21:33:23.889909 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.889818 4848 scope.go:122] "RemoveContainer" containerID="99739c29e97d34cb7dae40b0820ebf495474db9637a5116103479f887398b8ce" Oct 30 21:33:23.900569 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.900549 4848 scope.go:122] "RemoveContainer" containerID="99739c29e97d34cb7dae40b0820ebf495474db9637a5116103479f887398b8ce" Oct 30 21:33:23.900880 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:33:23.900862 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99739c29e97d34cb7dae40b0820ebf495474db9637a5116103479f887398b8ce\": container with ID starting with 99739c29e97d34cb7dae40b0820ebf495474db9637a5116103479f887398b8ce not found: ID does not exist" containerID="99739c29e97d34cb7dae40b0820ebf495474db9637a5116103479f887398b8ce" Oct 30 21:33:23.900960 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.900886 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99739c29e97d34cb7dae40b0820ebf495474db9637a5116103479f887398b8ce"} err="failed to get container status \"99739c29e97d34cb7dae40b0820ebf495474db9637a5116103479f887398b8ce\": rpc error: code = NotFound desc = could not find container \"99739c29e97d34cb7dae40b0820ebf495474db9637a5116103479f887398b8ce\": container with ID starting with 99739c29e97d34cb7dae40b0820ebf495474db9637a5116103479f887398b8ce not found: ID does not exist" Oct 30 21:33:23.912617 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.912565 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs"] Oct 30 21:33:23.916906 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:23.916886 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d95261-openshift-default-664cbbcc4f-z6trs"] Oct 30 21:33:24.931420 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:24.931386 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c705c9c-fef2-422b-b539-4007e28fedee" path="/var/lib/kubelet/pods/5c705c9c-fef2-422b-b539-4007e28fedee/volumes" Oct 30 21:33:31.980890 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:31.980857 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s"] Oct 30 21:33:31.981319 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:31.981046 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" podUID="06664297-01f4-4a6f-80de-309ab01eabf0" containerName="maas-api" containerID="cri-o://80f026436581c1633fb51c8f4ba6ea2275c2a3f88fec700a901cb58c4f5005bd" gracePeriod=30 Oct 30 21:33:32.255569 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.255538 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" Oct 30 21:33:32.347753 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.347719 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06664297-01f4-4a6f-80de-309ab01eabf0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06664297-01f4-4a6f-80de-309ab01eabf0-openshift-service-ca\") pod \"06664297-01f4-4a6f-80de-309ab01eabf0\" (UID: \"06664297-01f4-4a6f-80de-309ab01eabf0\") " Oct 30 21:33:32.347966 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.347758 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06664297-01f4-4a6f-80de-309ab01eabf0-kube-api-access-nzffj\" (UniqueName: \"kubernetes.io/projected/06664297-01f4-4a6f-80de-309ab01eabf0-kube-api-access-nzffj\") pod \"06664297-01f4-4a6f-80de-309ab01eabf0\" (UID: \"06664297-01f4-4a6f-80de-309ab01eabf0\") " Oct 30 21:33:32.347966 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.347852 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06664297-01f4-4a6f-80de-309ab01eabf0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06664297-01f4-4a6f-80de-309ab01eabf0-maas-api-tls\") pod \"06664297-01f4-4a6f-80de-309ab01eabf0\" (UID: \"06664297-01f4-4a6f-80de-309ab01eabf0\") " Oct 30 21:33:32.348046 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.348009 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06664297-01f4-4a6f-80de-309ab01eabf0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/06664297-01f4-4a6f-80de-309ab01eabf0-metrics-certs\") pod \"06664297-01f4-4a6f-80de-309ab01eabf0\" (UID: \"06664297-01f4-4a6f-80de-309ab01eabf0\") " Oct 30 21:33:32.348236 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.348206 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06664297-01f4-4a6f-80de-309ab01eabf0-openshift-service-ca" pod "06664297-01f4-4a6f-80de-309ab01eabf0" (UID: "06664297-01f4-4a6f-80de-309ab01eabf0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:32.348453 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.348433 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06664297-01f4-4a6f-80de-309ab01eabf0-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:32.350195 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.350166 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06664297-01f4-4a6f-80de-309ab01eabf0-metrics-certs" pod "06664297-01f4-4a6f-80de-309ab01eabf0" (UID: "06664297-01f4-4a6f-80de-309ab01eabf0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:32.350447 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.350430 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06664297-01f4-4a6f-80de-309ab01eabf0-kube-api-access-nzffj" pod "06664297-01f4-4a6f-80de-309ab01eabf0" (UID: "06664297-01f4-4a6f-80de-309ab01eabf0"). InnerVolumeSpecName "kube-api-access-nzffj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:32.350504 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.350444 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06664297-01f4-4a6f-80de-309ab01eabf0-maas-api-tls" pod "06664297-01f4-4a6f-80de-309ab01eabf0" (UID: "06664297-01f4-4a6f-80de-309ab01eabf0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:32.449296 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.449256 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzffj\" (UniqueName: \"kubernetes.io/projected/06664297-01f4-4a6f-80de-309ab01eabf0-kube-api-access-nzffj\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:32.449296 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.449290 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06664297-01f4-4a6f-80de-309ab01eabf0-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:32.449296 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.449304 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06664297-01f4-4a6f-80de-309ab01eabf0-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:32.932461 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.932424 4848 generic.go:356] "Generic (PLEG): container finished" podID="06664297-01f4-4a6f-80de-309ab01eabf0" containerID="80f026436581c1633fb51c8f4ba6ea2275c2a3f88fec700a901cb58c4f5005bd" exitCode=0 Oct 30 21:33:32.932673 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.932473 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" Oct 30 21:33:32.932673 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.932507 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" event={"ID":"06664297-01f4-4a6f-80de-309ab01eabf0","Type":"ContainerDied","Data":"80f026436581c1633fb51c8f4ba6ea2275c2a3f88fec700a901cb58c4f5005bd"} Oct 30 21:33:32.932673 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.932572 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s" event={"ID":"06664297-01f4-4a6f-80de-309ab01eabf0","Type":"ContainerDied","Data":"355b6e9f16182c339be183b778f760df1e3cb0a5af74897ed6a4cab320725e9f"} Oct 30 21:33:32.932673 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.932604 4848 scope.go:122] "RemoveContainer" containerID="80f026436581c1633fb51c8f4ba6ea2275c2a3f88fec700a901cb58c4f5005bd" Oct 30 21:33:32.945388 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.945362 4848 scope.go:122] "RemoveContainer" containerID="80f026436581c1633fb51c8f4ba6ea2275c2a3f88fec700a901cb58c4f5005bd" Oct 30 21:33:32.946024 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:33:32.945993 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80f026436581c1633fb51c8f4ba6ea2275c2a3f88fec700a901cb58c4f5005bd\": container with ID starting with 80f026436581c1633fb51c8f4ba6ea2275c2a3f88fec700a901cb58c4f5005bd not found: ID does not exist" containerID="80f026436581c1633fb51c8f4ba6ea2275c2a3f88fec700a901cb58c4f5005bd" Oct 30 21:33:32.946116 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.946034 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f026436581c1633fb51c8f4ba6ea2275c2a3f88fec700a901cb58c4f5005bd"} err="failed to get container status \"80f026436581c1633fb51c8f4ba6ea2275c2a3f88fec700a901cb58c4f5005bd\": rpc error: code = NotFound desc = could not find container \"80f026436581c1633fb51c8f4ba6ea2275c2a3f88fec700a901cb58c4f5005bd\": container with ID starting with 80f026436581c1633fb51c8f4ba6ea2275c2a3f88fec700a901cb58c4f5005bd not found: ID does not exist" Oct 30 21:33:32.959882 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.959861 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s"] Oct 30 21:33:32.964387 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:32.964365 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e53c37-79955684cc-x7n5s"] Oct 30 21:33:34.930632 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:34.930601 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06664297-01f4-4a6f-80de-309ab01eabf0" path="/var/lib/kubelet/pods/06664297-01f4-4a6f-80de-309ab01eabf0/volumes" Oct 30 21:33:39.916012 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:39.915979 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx"] Oct 30 21:33:39.916522 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:39.916240 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" podUID="d2715482-f623-43ef-93b8-c92d26626bc9" containerName="istio-proxy" containerID="cri-o://9d5b35a1d2a1254aa9c157f1761a9550e9a392da82f4b132db565a6d2c4df49b" gracePeriod=30 Oct 30 21:33:40.436744 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.436710 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl"] Oct 30 21:33:40.437296 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437274 4848 cpu_manager.go:414] "containerMap: removing container" podUID="a1bfc658-f4c9-4280-91e9-5081b0f9416e" containerName="istio-proxy" Oct 30 21:33:40.437388 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437302 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1bfc658-f4c9-4280-91e9-5081b0f9416e" containerName="istio-proxy" Oct 30 21:33:40.437388 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437322 4848 cpu_manager.go:414] "containerMap: removing container" podUID="28231432-c0bf-4c3b-879e-40acaceddeec" containerName="cleanup" Oct 30 21:33:40.437388 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437328 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28231432-c0bf-4c3b-879e-40acaceddeec" containerName="cleanup" Oct 30 21:33:40.437388 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437352 4848 cpu_manager.go:414] "containerMap: removing container" podUID="b7f2555a-89d9-4d58-89d5-f076b60af465" containerName="payload-pre-processing" Oct 30 21:33:40.437388 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437360 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f2555a-89d9-4d58-89d5-f076b60af465" containerName="payload-pre-processing" Oct 30 21:33:40.437388 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437379 4848 cpu_manager.go:414] "containerMap: removing container" podUID="06664297-01f4-4a6f-80de-309ab01eabf0" containerName="maas-api" Oct 30 21:33:40.437388 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437385 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06664297-01f4-4a6f-80de-309ab01eabf0" containerName="maas-api" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437400 4848 cpu_manager.go:414] "containerMap: removing container" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437407 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437417 4848 cpu_manager.go:414] "containerMap: removing container" podUID="3d385f07-6dae-4ca6-9e02-eb587a7d6374" containerName="payload-processing" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437424 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d385f07-6dae-4ca6-9e02-eb587a7d6374" containerName="payload-processing" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437432 4848 cpu_manager.go:414] "containerMap: removing container" podUID="a3aee534-70d3-4567-8a70-063a4e684526" containerName="maas-api" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437439 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3aee534-70d3-4567-8a70-063a4e684526" containerName="maas-api" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437447 4848 cpu_manager.go:414] "containerMap: removing container" podUID="7c946890-dd61-47f5-8c84-ef70e09b2da4" containerName="cleanup" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437453 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c946890-dd61-47f5-8c84-ef70e09b2da4" containerName="cleanup" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437462 4848 cpu_manager.go:414] "containerMap: removing container" podUID="5c705c9c-fef2-422b-b539-4007e28fedee" containerName="istio-proxy" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437468 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c705c9c-fef2-422b-b539-4007e28fedee" containerName="istio-proxy" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437480 4848 cpu_manager.go:414] "containerMap: removing container" podUID="6867e706-6187-4821-ae3c-141f443718a5" containerName="payload-pre-processing" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437487 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6867e706-6187-4821-ae3c-141f443718a5" containerName="payload-pre-processing" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437497 4848 cpu_manager.go:414] "containerMap: removing container" podUID="691eb0e8-a109-4d75-90c5-d7a7bedd9be0" containerName="cleanup" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437503 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="691eb0e8-a109-4d75-90c5-d7a7bedd9be0" containerName="cleanup" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437517 4848 cpu_manager.go:414] "containerMap: removing container" podUID="e57add2c-5ccf-47eb-8adb-d5852289b111" containerName="istio-proxy" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437524 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e57add2c-5ccf-47eb-8adb-d5852289b111" containerName="istio-proxy" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437534 4848 cpu_manager.go:414] "containerMap: removing container" podUID="a88a7285-af5c-4693-93fb-8792d1f8959c" containerName="revoke-keys" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437540 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a88a7285-af5c-4693-93fb-8792d1f8959c" containerName="revoke-keys" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437556 4848 cpu_manager.go:414] "containerMap: removing container" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437562 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437575 4848 cpu_manager.go:414] "containerMap: removing container" podUID="e57add2c-5ccf-47eb-8adb-d5852289b111" containerName="istio-proxy" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437581 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e57add2c-5ccf-47eb-8adb-d5852289b111" containerName="istio-proxy" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437678 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="e57add2c-5ccf-47eb-8adb-d5852289b111" containerName="istio-proxy" Oct 30 21:33:40.437682 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437690 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="a88a7285-af5c-4693-93fb-8792d1f8959c" containerName="revoke-keys" Oct 30 21:33:40.438619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437700 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1bfc658-f4c9-4280-91e9-5081b0f9416e" containerName="istio-proxy" Oct 30 21:33:40.438619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437709 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6f0c012-836a-4de1-b9a8-2acd35ec9ce8" containerName="istio-proxy" Oct 30 21:33:40.438619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437718 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc3af69c-3d4e-4aab-98af-9082f94ace11" containerName="main" Oct 30 21:33:40.438619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437727 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="e57add2c-5ccf-47eb-8adb-d5852289b111" containerName="istio-proxy" Oct 30 21:33:40.438619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437735 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1bfc658-f4c9-4280-91e9-5081b0f9416e" containerName="istio-proxy" Oct 30 21:33:40.438619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437743 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="28231432-c0bf-4c3b-879e-40acaceddeec" containerName="cleanup" Oct 30 21:33:40.438619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437753 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f2555a-89d9-4d58-89d5-f076b60af465" containerName="payload-pre-processing" Oct 30 21:33:40.438619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437782 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="06664297-01f4-4a6f-80de-309ab01eabf0" containerName="maas-api" Oct 30 21:33:40.438619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437790 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c4c39fa-0000-40ad-8f24-53ae45fe0f7e" containerName="istio-proxy" Oct 30 21:33:40.438619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437799 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="130328d3-b4f3-4123-9116-c5e462b0c8dc" containerName="main" Oct 30 21:33:40.438619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437808 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d385f07-6dae-4ca6-9e02-eb587a7d6374" containerName="payload-processing" Oct 30 21:33:40.438619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437816 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3aee534-70d3-4567-8a70-063a4e684526" containerName="maas-api" Oct 30 21:33:40.438619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437824 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c705c9c-fef2-422b-b539-4007e28fedee" containerName="istio-proxy" Oct 30 21:33:40.438619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437831 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="6867e706-6187-4821-ae3c-141f443718a5" containerName="payload-pre-processing" Oct 30 21:33:40.438619 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.437839 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="691eb0e8-a109-4d75-90c5-d7a7bedd9be0" containerName="cleanup" Oct 30 21:33:40.441269 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.441253 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl" Oct 30 21:33:40.447572 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.447522 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl"] Oct 30 21:33:40.468071 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.468047 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:33:40.468647 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.468623 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\"" Oct 30 21:33:40.536221 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.536190 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl\" (UID: \"29a22e0f-f4d7-4fe9-8a25-49e8e9284422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl" Oct 30 21:33:40.536357 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.536246 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl\" (UID: \"29a22e0f-f4d7-4fe9-8a25-49e8e9284422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl" Oct 30 21:33:40.536406 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.536353 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fw4q\" (UniqueName: \"kubernetes.io/projected/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-kube-api-access-9fw4q\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl\" (UID: \"29a22e0f-f4d7-4fe9-8a25-49e8e9284422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl" Oct 30 21:33:40.637878 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.637843 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl\" (UID: \"29a22e0f-f4d7-4fe9-8a25-49e8e9284422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl" Oct 30 21:33:40.638030 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.637894 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fw4q\" (UniqueName: \"kubernetes.io/projected/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-kube-api-access-9fw4q\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl\" (UID: \"29a22e0f-f4d7-4fe9-8a25-49e8e9284422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl" Oct 30 21:33:40.638030 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.637958 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl\" (UID: \"29a22e0f-f4d7-4fe9-8a25-49e8e9284422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl" Oct 30 21:33:40.638545 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.638520 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl\" (UID: \"29a22e0f-f4d7-4fe9-8a25-49e8e9284422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl" Oct 30 21:33:40.640219 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.640194 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl\" (UID: \"29a22e0f-f4d7-4fe9-8a25-49e8e9284422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl" Oct 30 21:33:40.649257 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.649230 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fw4q\" (UniqueName: \"kubernetes.io/projected/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-kube-api-access-9fw4q\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl\" (UID: \"29a22e0f-f4d7-4fe9-8a25-49e8e9284422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl" Oct 30 21:33:40.757482 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:40.757392 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl" Oct 30 21:33:41.030447 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:41.030387 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl"] Oct 30 21:33:41.032942 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:33:41.032911 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a22e0f_f4d7_4fe9_8a25_49e8e9284422.slice/crio-18264d777079dc10c752564c796f8700f73b578f6ea4b897652abdb9377271fb WatchSource:0}: Error finding container 18264d777079dc10c752564c796f8700f73b578f6ea4b897652abdb9377271fb: Status 404 returned error can't find the container with id 18264d777079dc10c752564c796f8700f73b578f6ea4b897652abdb9377271fb Oct 30 21:33:41.973076 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:41.973038 4848 generic.go:356] "Generic (PLEG): container finished" podID="29a22e0f-f4d7-4fe9-8a25-49e8e9284422" containerID="87ad8dc74b05dbf0e7733def337ce28cd29d95f797c8e5f9c2b482054ca9eb26" exitCode=0 Oct 30 21:33:41.973257 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:41.973097 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl" event={"ID":"29a22e0f-f4d7-4fe9-8a25-49e8e9284422","Type":"ContainerDied","Data":"87ad8dc74b05dbf0e7733def337ce28cd29d95f797c8e5f9c2b482054ca9eb26"} Oct 30 21:33:41.973257 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:41.973150 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl" event={"ID":"29a22e0f-f4d7-4fe9-8a25-49e8e9284422","Type":"ContainerStarted","Data":"18264d777079dc10c752564c796f8700f73b578f6ea4b897652abdb9377271fb"} Oct 30 21:33:43.120216 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:43.120192 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl" Oct 30 21:33:43.263184 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:43.263100 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-maas-api-xks-ca\") pod \"29a22e0f-f4d7-4fe9-8a25-49e8e9284422\" (UID: \"29a22e0f-f4d7-4fe9-8a25-49e8e9284422\") " Oct 30 21:33:43.263184 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:43.263177 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-maas-api-openshift-ca\") pod \"29a22e0f-f4d7-4fe9-8a25-49e8e9284422\" (UID: \"29a22e0f-f4d7-4fe9-8a25-49e8e9284422\") " Oct 30 21:33:43.263423 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:43.263242 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-kube-api-access-9fw4q\" (UniqueName: \"kubernetes.io/projected/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-kube-api-access-9fw4q\") pod \"29a22e0f-f4d7-4fe9-8a25-49e8e9284422\" (UID: \"29a22e0f-f4d7-4fe9-8a25-49e8e9284422\") " Oct 30 21:33:43.263620 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:43.263591 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-maas-api-openshift-ca" pod "29a22e0f-f4d7-4fe9-8a25-49e8e9284422" (UID: "29a22e0f-f4d7-4fe9-8a25-49e8e9284422"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:43.265275 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:43.265250 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-kube-api-access-9fw4q" pod "29a22e0f-f4d7-4fe9-8a25-49e8e9284422" (UID: "29a22e0f-f4d7-4fe9-8a25-49e8e9284422"). InnerVolumeSpecName "kube-api-access-9fw4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:43.265275 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:43.265260 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-maas-api-xks-ca" pod "29a22e0f-f4d7-4fe9-8a25-49e8e9284422" (UID: "29a22e0f-f4d7-4fe9-8a25-49e8e9284422"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:43.364667 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:43.364632 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-maas-api-openshift-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:43.364667 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:43.364662 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fw4q\" (UniqueName: \"kubernetes.io/projected/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-kube-api-access-9fw4q\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:43.364667 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:43.364672 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29a22e0f-f4d7-4fe9-8a25-49e8e9284422-maas-api-xks-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:43.983119 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:43.983092 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl" Oct 30 21:33:43.983119 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:43.983108 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl" event={"ID":"29a22e0f-f4d7-4fe9-8a25-49e8e9284422","Type":"ContainerDied","Data":"18264d777079dc10c752564c796f8700f73b578f6ea4b897652abdb9377271fb"} Oct 30 21:33:43.983357 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:43.983156 4848 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18264d777079dc10c752564c796f8700f73b578f6ea4b897652abdb9377271fb" Oct 30 21:33:45.164609 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.164585 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:33:45.284863 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.284777 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2715482-f623-43ef-93b8-c92d26626bc9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2715482-f623-43ef-93b8-c92d26626bc9-istiod-ca-cert\") pod \"d2715482-f623-43ef-93b8-c92d26626bc9\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " Oct 30 21:33:45.284863 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.284859 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-istio-envoy\") pod \"d2715482-f623-43ef-93b8-c92d26626bc9\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " Oct 30 21:33:45.285106 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.284883 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-workload-socket\") pod \"d2715482-f623-43ef-93b8-c92d26626bc9\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " Oct 30 21:33:45.285106 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.284947 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-workload-certs\") pod \"d2715482-f623-43ef-93b8-c92d26626bc9\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " Oct 30 21:33:45.285106 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.285003 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-istio-data\") pod \"d2715482-f623-43ef-93b8-c92d26626bc9\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " Oct 30 21:33:45.285106 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.285032 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2715482-f623-43ef-93b8-c92d26626bc9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2715482-f623-43ef-93b8-c92d26626bc9-istio-podinfo\") pod \"d2715482-f623-43ef-93b8-c92d26626bc9\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " Oct 30 21:33:45.285106 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.285089 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2715482-f623-43ef-93b8-c92d26626bc9-istio-token\" (UniqueName: \"kubernetes.io/projected/d2715482-f623-43ef-93b8-c92d26626bc9-istio-token\") pod \"d2715482-f623-43ef-93b8-c92d26626bc9\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " Oct 30 21:33:45.285364 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.285095 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2715482-f623-43ef-93b8-c92d26626bc9-istiod-ca-cert" pod "d2715482-f623-43ef-93b8-c92d26626bc9" (UID: "d2715482-f623-43ef-93b8-c92d26626bc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:45.285364 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.285122 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2715482-f623-43ef-93b8-c92d26626bc9-kube-api-access-p659p\" (UniqueName: \"kubernetes.io/projected/d2715482-f623-43ef-93b8-c92d26626bc9-kube-api-access-p659p\") pod \"d2715482-f623-43ef-93b8-c92d26626bc9\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " Oct 30 21:33:45.285364 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.285172 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-credential-socket\") pod \"d2715482-f623-43ef-93b8-c92d26626bc9\" (UID: \"d2715482-f623-43ef-93b8-c92d26626bc9\") " Oct 30 21:33:45.285364 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.285261 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-workload-socket" pod "d2715482-f623-43ef-93b8-c92d26626bc9" (UID: "d2715482-f623-43ef-93b8-c92d26626bc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:45.285564 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.285402 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-workload-certs" pod "d2715482-f623-43ef-93b8-c92d26626bc9" (UID: "d2715482-f623-43ef-93b8-c92d26626bc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:45.285564 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.285527 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:45.285564 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.285548 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:45.285564 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.285562 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2715482-f623-43ef-93b8-c92d26626bc9-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:45.285818 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.285795 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-istio-data" pod "d2715482-f623-43ef-93b8-c92d26626bc9" (UID: "d2715482-f623-43ef-93b8-c92d26626bc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:45.286062 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.286034 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-credential-socket" pod "d2715482-f623-43ef-93b8-c92d26626bc9" (UID: "d2715482-f623-43ef-93b8-c92d26626bc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:45.287596 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.287568 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2715482-f623-43ef-93b8-c92d26626bc9-istio-podinfo" pod "d2715482-f623-43ef-93b8-c92d26626bc9" (UID: "d2715482-f623-43ef-93b8-c92d26626bc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:33:45.287677 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.287637 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2715482-f623-43ef-93b8-c92d26626bc9-kube-api-access-p659p" pod "d2715482-f623-43ef-93b8-c92d26626bc9" (UID: "d2715482-f623-43ef-93b8-c92d26626bc9"). InnerVolumeSpecName "kube-api-access-p659p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:45.287737 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.287721 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2715482-f623-43ef-93b8-c92d26626bc9-istio-token" pod "d2715482-f623-43ef-93b8-c92d26626bc9" (UID: "d2715482-f623-43ef-93b8-c92d26626bc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:45.287817 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.287735 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-istio-envoy" pod "d2715482-f623-43ef-93b8-c92d26626bc9" (UID: "d2715482-f623-43ef-93b8-c92d26626bc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:45.386232 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.386202 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:45.386232 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.386229 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2715482-f623-43ef-93b8-c92d26626bc9-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:45.386232 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.386239 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2715482-f623-43ef-93b8-c92d26626bc9-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:45.386469 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.386249 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p659p\" (UniqueName: \"kubernetes.io/projected/d2715482-f623-43ef-93b8-c92d26626bc9-kube-api-access-p659p\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:45.386469 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.386260 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:45.386469 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.386268 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2715482-f623-43ef-93b8-c92d26626bc9-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:45.994441 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.994406 4848 generic.go:356] "Generic (PLEG): container finished" podID="d2715482-f623-43ef-93b8-c92d26626bc9" containerID="9d5b35a1d2a1254aa9c157f1761a9550e9a392da82f4b132db565a6d2c4df49b" exitCode=0 Oct 30 21:33:45.994643 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.994457 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" Oct 30 21:33:45.994643 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.994457 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" event={"ID":"d2715482-f623-43ef-93b8-c92d26626bc9","Type":"ContainerDied","Data":"9d5b35a1d2a1254aa9c157f1761a9550e9a392da82f4b132db565a6d2c4df49b"} Oct 30 21:33:45.994643 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.994583 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx" event={"ID":"d2715482-f623-43ef-93b8-c92d26626bc9","Type":"ContainerDied","Data":"c44fd859850cdd96e619729f8dab173480e9304650ecb2f254f986bf07243b9f"} Oct 30 21:33:45.994643 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:45.994608 4848 scope.go:122] "RemoveContainer" containerID="9d5b35a1d2a1254aa9c157f1761a9550e9a392da82f4b132db565a6d2c4df49b" Oct 30 21:33:46.004212 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:46.004187 4848 scope.go:122] "RemoveContainer" containerID="9d5b35a1d2a1254aa9c157f1761a9550e9a392da82f4b132db565a6d2c4df49b" Oct 30 21:33:46.004521 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:33:46.004491 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5b35a1d2a1254aa9c157f1761a9550e9a392da82f4b132db565a6d2c4df49b\": container with ID starting with 9d5b35a1d2a1254aa9c157f1761a9550e9a392da82f4b132db565a6d2c4df49b not found: ID does not exist" containerID="9d5b35a1d2a1254aa9c157f1761a9550e9a392da82f4b132db565a6d2c4df49b" Oct 30 21:33:46.004607 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:46.004533 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5b35a1d2a1254aa9c157f1761a9550e9a392da82f4b132db565a6d2c4df49b"} err="failed to get container status \"9d5b35a1d2a1254aa9c157f1761a9550e9a392da82f4b132db565a6d2c4df49b\": rpc error: code = NotFound desc = could not find container \"9d5b35a1d2a1254aa9c157f1761a9550e9a392da82f4b132db565a6d2c4df49b\": container with ID starting with 9d5b35a1d2a1254aa9c157f1761a9550e9a392da82f4b132db565a6d2c4df49b not found: ID does not exist" Oct 30 21:33:46.019519 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:46.019490 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx"] Oct 30 21:33:46.026418 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:46.026392 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-e53c37-openshift-default-84d754496f-b5smx"] Oct 30 21:33:46.930382 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:46.930352 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2715482-f623-43ef-93b8-c92d26626bc9" path="/var/lib/kubelet/pods/d2715482-f623-43ef-93b8-c92d26626bc9/volumes" Oct 30 21:33:51.888942 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:51.888907 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f"] Oct 30 21:33:51.889759 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:51.889091 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" podUID="01342f11-19ac-41fe-87e2-d07bc8c3c4ba" containerName="maas-api" containerID="cri-o://dd525585077086397541ff2dec2009d313646dc9bf8fe2d32da6f2c6681c6f73" gracePeriod=30 Oct 30 21:33:52.025526 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:52.025495 4848 generic.go:356] "Generic (PLEG): container finished" podID="01342f11-19ac-41fe-87e2-d07bc8c3c4ba" containerID="dd525585077086397541ff2dec2009d313646dc9bf8fe2d32da6f2c6681c6f73" exitCode=0 Oct 30 21:33:52.025691 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:52.025566 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" event={"ID":"01342f11-19ac-41fe-87e2-d07bc8c3c4ba","Type":"ContainerDied","Data":"dd525585077086397541ff2dec2009d313646dc9bf8fe2d32da6f2c6681c6f73"} Oct 30 21:33:52.149560 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:52.149500 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" Oct 30 21:33:52.158871 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:52.158842 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl"] Oct 30 21:33:52.161165 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:52.161135 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-321ccd-wwlbl"] Oct 30 21:33:52.249438 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:52.249398 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-metrics-certs\") pod \"01342f11-19ac-41fe-87e2-d07bc8c3c4ba\" (UID: \"01342f11-19ac-41fe-87e2-d07bc8c3c4ba\") " Oct 30 21:33:52.249651 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:52.249455 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-maas-api-tls\") pod \"01342f11-19ac-41fe-87e2-d07bc8c3c4ba\" (UID: \"01342f11-19ac-41fe-87e2-d07bc8c3c4ba\") " Oct 30 21:33:52.249651 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:52.249482 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-kube-api-access-q8mwb\" (UniqueName: \"kubernetes.io/projected/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-kube-api-access-q8mwb\") pod \"01342f11-19ac-41fe-87e2-d07bc8c3c4ba\" (UID: \"01342f11-19ac-41fe-87e2-d07bc8c3c4ba\") " Oct 30 21:33:52.249651 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:52.249520 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-openshift-service-ca\") pod \"01342f11-19ac-41fe-87e2-d07bc8c3c4ba\" (UID: \"01342f11-19ac-41fe-87e2-d07bc8c3c4ba\") " Oct 30 21:33:52.249991 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:52.249960 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-openshift-service-ca" pod "01342f11-19ac-41fe-87e2-d07bc8c3c4ba" (UID: "01342f11-19ac-41fe-87e2-d07bc8c3c4ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:52.251904 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:52.251875 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-kube-api-access-q8mwb" pod "01342f11-19ac-41fe-87e2-d07bc8c3c4ba" (UID: "01342f11-19ac-41fe-87e2-d07bc8c3c4ba"). InnerVolumeSpecName "kube-api-access-q8mwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:52.252137 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:52.252116 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-metrics-certs" pod "01342f11-19ac-41fe-87e2-d07bc8c3c4ba" (UID: "01342f11-19ac-41fe-87e2-d07bc8c3c4ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:52.252252 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:52.252228 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-maas-api-tls" pod "01342f11-19ac-41fe-87e2-d07bc8c3c4ba" (UID: "01342f11-19ac-41fe-87e2-d07bc8c3c4ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:52.350477 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:52.350439 4848 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-metrics-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:52.350477 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:52.350473 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-maas-api-tls\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:52.350685 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:52.350489 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8mwb\" (UniqueName: \"kubernetes.io/projected/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-kube-api-access-q8mwb\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:52.350685 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:52.350506 4848 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01342f11-19ac-41fe-87e2-d07bc8c3c4ba-openshift-service-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:33:52.511418 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:33:52.511378 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:33:52.932187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:52.932148 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29a22e0f-f4d7-4fe9-8a25-49e8e9284422" path="/var/lib/kubelet/pods/29a22e0f-f4d7-4fe9-8a25-49e8e9284422/volumes" Oct 30 21:33:53.033116 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:53.033081 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" Oct 30 21:33:53.033116 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:53.033097 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f" event={"ID":"01342f11-19ac-41fe-87e2-d07bc8c3c4ba","Type":"ContainerDied","Data":"9e4857a620a4d64caf52ce9d57e627c3f005c67cca73f6b53d8fe0b32627fc0a"} Oct 30 21:33:53.033391 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:53.033166 4848 scope.go:122] "RemoveContainer" containerID="dd525585077086397541ff2dec2009d313646dc9bf8fe2d32da6f2c6681c6f73" Oct 30 21:33:53.052727 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:53.052698 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f"] Oct 30 21:33:53.057681 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:53.057654 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-321ccd-57477787b4-pdr8f"] Oct 30 21:33:54.931263 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:33:54.931232 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01342f11-19ac-41fe-87e2-d07bc8c3c4ba" path="/var/lib/kubelet/pods/01342f11-19ac-41fe-87e2-d07bc8c3c4ba/volumes" Oct 30 21:34:00.301418 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.301380 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l"] Oct 30 21:34:00.301893 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.301828 4848 cpu_manager.go:414] "containerMap: removing container" podUID="d2715482-f623-43ef-93b8-c92d26626bc9" containerName="istio-proxy" Oct 30 21:34:00.301893 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.301844 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2715482-f623-43ef-93b8-c92d26626bc9" containerName="istio-proxy" Oct 30 21:34:00.301893 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.301876 4848 cpu_manager.go:414] "containerMap: removing container" podUID="29a22e0f-f4d7-4fe9-8a25-49e8e9284422" containerName="revoke-keys" Oct 30 21:34:00.301893 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.301881 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a22e0f-f4d7-4fe9-8a25-49e8e9284422" containerName="revoke-keys" Oct 30 21:34:00.301893 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.301890 4848 cpu_manager.go:414] "containerMap: removing container" podUID="01342f11-19ac-41fe-87e2-d07bc8c3c4ba" containerName="maas-api" Oct 30 21:34:00.301893 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.301896 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01342f11-19ac-41fe-87e2-d07bc8c3c4ba" containerName="maas-api" Oct 30 21:34:00.302156 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.301903 4848 cpu_manager.go:414] "containerMap: removing container" podUID="a1bfc658-f4c9-4280-91e9-5081b0f9416e" containerName="istio-proxy" Oct 30 21:34:00.302156 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.301908 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1bfc658-f4c9-4280-91e9-5081b0f9416e" containerName="istio-proxy" Oct 30 21:34:00.302156 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.301981 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2715482-f623-43ef-93b8-c92d26626bc9" containerName="istio-proxy" Oct 30 21:34:00.302156 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.301999 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a22e0f-f4d7-4fe9-8a25-49e8e9284422" containerName="revoke-keys" Oct 30 21:34:00.302156 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.302006 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="01342f11-19ac-41fe-87e2-d07bc8c3c4ba" containerName="maas-api" Oct 30 21:34:00.306856 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.306835 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l" Oct 30 21:34:00.313505 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.313385 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l"] Oct 30 21:34:00.337092 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.337057 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:34:00.337380 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.337357 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\"" Oct 30 21:34:00.419265 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.419226 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/817ceb47-dc06-4c56-964a-d636f27dfefe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l\" (UID: \"817ceb47-dc06-4c56-964a-d636f27dfefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l" Oct 30 21:34:00.419433 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.419269 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzxsm\" (UniqueName: \"kubernetes.io/projected/817ceb47-dc06-4c56-964a-d636f27dfefe-kube-api-access-rzxsm\") pod \"maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l\" (UID: \"817ceb47-dc06-4c56-964a-d636f27dfefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l" Oct 30 21:34:00.419433 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.419394 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/817ceb47-dc06-4c56-964a-d636f27dfefe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l\" (UID: \"817ceb47-dc06-4c56-964a-d636f27dfefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l" Oct 30 21:34:00.520551 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.520515 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/817ceb47-dc06-4c56-964a-d636f27dfefe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l\" (UID: \"817ceb47-dc06-4c56-964a-d636f27dfefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l" Oct 30 21:34:00.520551 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.520552 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzxsm\" (UniqueName: \"kubernetes.io/projected/817ceb47-dc06-4c56-964a-d636f27dfefe-kube-api-access-rzxsm\") pod \"maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l\" (UID: \"817ceb47-dc06-4c56-964a-d636f27dfefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l" Oct 30 21:34:00.520761 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.520599 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/817ceb47-dc06-4c56-964a-d636f27dfefe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l\" (UID: \"817ceb47-dc06-4c56-964a-d636f27dfefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l" Oct 30 21:34:00.521204 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.521174 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/817ceb47-dc06-4c56-964a-d636f27dfefe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l\" (UID: \"817ceb47-dc06-4c56-964a-d636f27dfefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l" Oct 30 21:34:00.523030 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.523000 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/817ceb47-dc06-4c56-964a-d636f27dfefe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l\" (UID: \"817ceb47-dc06-4c56-964a-d636f27dfefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l" Oct 30 21:34:00.529358 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.529332 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzxsm\" (UniqueName: \"kubernetes.io/projected/817ceb47-dc06-4c56-964a-d636f27dfefe-kube-api-access-rzxsm\") pod \"maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l\" (UID: \"817ceb47-dc06-4c56-964a-d636f27dfefe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l" Oct 30 21:34:00.625034 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.624947 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l" Oct 30 21:34:00.910951 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:00.910872 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l"] Oct 30 21:34:00.914527 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:34:00.914501 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817ceb47_dc06_4c56_964a_d636f27dfefe.slice/crio-e57236614b7abdeabbf78ab7c5ce27f99aeb03da261cf84f88048d1371b8e54d WatchSource:0}: Error finding container e57236614b7abdeabbf78ab7c5ce27f99aeb03da261cf84f88048d1371b8e54d: Status 404 returned error can't find the container with id e57236614b7abdeabbf78ab7c5ce27f99aeb03da261cf84f88048d1371b8e54d Oct 30 21:34:01.073831 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:01.073793 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l" event={"ID":"817ceb47-dc06-4c56-964a-d636f27dfefe","Type":"ContainerStarted","Data":"e57236614b7abdeabbf78ab7c5ce27f99aeb03da261cf84f88048d1371b8e54d"} Oct 30 21:34:02.079800 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:02.079704 4848 generic.go:356] "Generic (PLEG): container finished" podID="817ceb47-dc06-4c56-964a-d636f27dfefe" containerID="4acb5d597108e88d41c3e37d790368508ad847d66e18e6b09c8a5af74ce86ceb" exitCode=0 Oct 30 21:34:02.080175 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:02.079786 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l" event={"ID":"817ceb47-dc06-4c56-964a-d636f27dfefe","Type":"ContainerDied","Data":"4acb5d597108e88d41c3e37d790368508ad847d66e18e6b09c8a5af74ce86ceb"} Oct 30 21:34:03.222789 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:03.222743 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l" Oct 30 21:34:03.250266 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:03.250238 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/817ceb47-dc06-4c56-964a-d636f27dfefe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/817ceb47-dc06-4c56-964a-d636f27dfefe-maas-api-openshift-ca\") pod \"817ceb47-dc06-4c56-964a-d636f27dfefe\" (UID: \"817ceb47-dc06-4c56-964a-d636f27dfefe\") " Oct 30 21:34:03.250420 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:03.250382 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/817ceb47-dc06-4c56-964a-d636f27dfefe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/817ceb47-dc06-4c56-964a-d636f27dfefe-maas-api-xks-ca\") pod \"817ceb47-dc06-4c56-964a-d636f27dfefe\" (UID: \"817ceb47-dc06-4c56-964a-d636f27dfefe\") " Oct 30 21:34:03.250489 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:03.250470 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/817ceb47-dc06-4c56-964a-d636f27dfefe-kube-api-access-rzxsm\" (UniqueName: \"kubernetes.io/projected/817ceb47-dc06-4c56-964a-d636f27dfefe-kube-api-access-rzxsm\") pod \"817ceb47-dc06-4c56-964a-d636f27dfefe\" (UID: \"817ceb47-dc06-4c56-964a-d636f27dfefe\") " Oct 30 21:34:03.250553 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:03.250521 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/817ceb47-dc06-4c56-964a-d636f27dfefe-maas-api-openshift-ca" pod "817ceb47-dc06-4c56-964a-d636f27dfefe" (UID: "817ceb47-dc06-4c56-964a-d636f27dfefe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:34:03.250836 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:03.250815 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/817ceb47-dc06-4c56-964a-d636f27dfefe-maas-api-openshift-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:03.252609 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:03.252587 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/817ceb47-dc06-4c56-964a-d636f27dfefe-kube-api-access-rzxsm" pod "817ceb47-dc06-4c56-964a-d636f27dfefe" (UID: "817ceb47-dc06-4c56-964a-d636f27dfefe"). InnerVolumeSpecName "kube-api-access-rzxsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:03.252721 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:03.252620 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/817ceb47-dc06-4c56-964a-d636f27dfefe-maas-api-xks-ca" pod "817ceb47-dc06-4c56-964a-d636f27dfefe" (UID: "817ceb47-dc06-4c56-964a-d636f27dfefe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:03.351378 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:03.351295 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzxsm\" (UniqueName: \"kubernetes.io/projected/817ceb47-dc06-4c56-964a-d636f27dfefe-kube-api-access-rzxsm\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:03.351378 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:03.351328 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/817ceb47-dc06-4c56-964a-d636f27dfefe-maas-api-xks-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:04.090737 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:04.090709 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l" Oct 30 21:34:04.090930 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:04.090702 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l" event={"ID":"817ceb47-dc06-4c56-964a-d636f27dfefe","Type":"ContainerDied","Data":"e57236614b7abdeabbf78ab7c5ce27f99aeb03da261cf84f88048d1371b8e54d"} Oct 30 21:34:04.090930 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:04.090837 4848 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e57236614b7abdeabbf78ab7c5ce27f99aeb03da261cf84f88048d1371b8e54d" Oct 30 21:34:11.423494 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.423452 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b312f00-w9p96"] Oct 30 21:34:11.423938 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.423910 4848 cpu_manager.go:414] "containerMap: removing container" podUID="817ceb47-dc06-4c56-964a-d636f27dfefe" containerName="revoke-keys" Oct 30 21:34:11.423938 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.423925 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="817ceb47-dc06-4c56-964a-d636f27dfefe" containerName="revoke-keys" Oct 30 21:34:11.424031 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.423989 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="817ceb47-dc06-4c56-964a-d636f27dfefe" containerName="revoke-keys" Oct 30 21:34:11.426800 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.426783 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78594f64-9wwpt"] Oct 30 21:34:11.426905 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.426883 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b312f00-w9p96" Oct 30 21:34:11.430237 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.430216 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78594f64-9wwpt" Oct 30 21:34:11.436003 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.435980 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:34:11.436120 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.436031 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kjd66\"" Oct 30 21:34:11.443083 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.443055 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78594f64-9wwpt"] Oct 30 21:34:11.449814 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.449789 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b312f00-w9p96"] Oct 30 21:34:11.515406 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.515367 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708051cf-4bca-453c-8072-77d6a0ceb89d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78594f64-9wwpt\" (UID: \"708051cf-4bca-453c-8072-77d6a0ceb89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78594f64-9wwpt" Oct 30 21:34:11.515406 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.515417 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqlsv\" (UniqueName: \"kubernetes.io/projected/3f908467-b845-4a03-849d-4ab0c9ef698b-kube-api-access-gqlsv\") pod \"maas-api-revoke-sub-0b312f00-w9p96\" (UID: \"3f908467-b845-4a03-849d-4ab0c9ef698b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b312f00-w9p96" Oct 30 21:34:11.515726 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.515456 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9xq8\" (UniqueName: \"kubernetes.io/projected/708051cf-4bca-453c-8072-77d6a0ceb89d-kube-api-access-s9xq8\") pod \"maas-api-revoke-sub-78594f64-9wwpt\" (UID: \"708051cf-4bca-453c-8072-77d6a0ceb89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78594f64-9wwpt" Oct 30 21:34:11.515726 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.515561 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708051cf-4bca-453c-8072-77d6a0ceb89d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78594f64-9wwpt\" (UID: \"708051cf-4bca-453c-8072-77d6a0ceb89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78594f64-9wwpt" Oct 30 21:34:11.515726 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.515636 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f908467-b845-4a03-849d-4ab0c9ef698b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b312f00-w9p96\" (UID: \"3f908467-b845-4a03-849d-4ab0c9ef698b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b312f00-w9p96" Oct 30 21:34:11.515726 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.515659 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f908467-b845-4a03-849d-4ab0c9ef698b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b312f00-w9p96\" (UID: \"3f908467-b845-4a03-849d-4ab0c9ef698b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b312f00-w9p96" Oct 30 21:34:11.617106 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.617067 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708051cf-4bca-453c-8072-77d6a0ceb89d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78594f64-9wwpt\" (UID: \"708051cf-4bca-453c-8072-77d6a0ceb89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78594f64-9wwpt" Oct 30 21:34:11.617320 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.617129 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f908467-b845-4a03-849d-4ab0c9ef698b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b312f00-w9p96\" (UID: \"3f908467-b845-4a03-849d-4ab0c9ef698b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b312f00-w9p96" Oct 30 21:34:11.617320 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.617165 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f908467-b845-4a03-849d-4ab0c9ef698b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b312f00-w9p96\" (UID: \"3f908467-b845-4a03-849d-4ab0c9ef698b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b312f00-w9p96" Oct 30 21:34:11.617320 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.617268 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708051cf-4bca-453c-8072-77d6a0ceb89d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78594f64-9wwpt\" (UID: \"708051cf-4bca-453c-8072-77d6a0ceb89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78594f64-9wwpt" Oct 30 21:34:11.617320 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.617293 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqlsv\" (UniqueName: \"kubernetes.io/projected/3f908467-b845-4a03-849d-4ab0c9ef698b-kube-api-access-gqlsv\") pod \"maas-api-revoke-sub-0b312f00-w9p96\" (UID: \"3f908467-b845-4a03-849d-4ab0c9ef698b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b312f00-w9p96" Oct 30 21:34:11.617544 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.617326 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9xq8\" (UniqueName: \"kubernetes.io/projected/708051cf-4bca-453c-8072-77d6a0ceb89d-kube-api-access-s9xq8\") pod \"maas-api-revoke-sub-78594f64-9wwpt\" (UID: \"708051cf-4bca-453c-8072-77d6a0ceb89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78594f64-9wwpt" Oct 30 21:34:11.617913 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.617887 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708051cf-4bca-453c-8072-77d6a0ceb89d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78594f64-9wwpt\" (UID: \"708051cf-4bca-453c-8072-77d6a0ceb89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78594f64-9wwpt" Oct 30 21:34:11.618100 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.617943 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f908467-b845-4a03-849d-4ab0c9ef698b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b312f00-w9p96\" (UID: \"3f908467-b845-4a03-849d-4ab0c9ef698b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b312f00-w9p96" Oct 30 21:34:11.619697 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.619671 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f908467-b845-4a03-849d-4ab0c9ef698b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b312f00-w9p96\" (UID: \"3f908467-b845-4a03-849d-4ab0c9ef698b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b312f00-w9p96" Oct 30 21:34:11.620002 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.619980 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708051cf-4bca-453c-8072-77d6a0ceb89d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78594f64-9wwpt\" (UID: \"708051cf-4bca-453c-8072-77d6a0ceb89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78594f64-9wwpt" Oct 30 21:34:11.625607 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.625586 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9xq8\" (UniqueName: \"kubernetes.io/projected/708051cf-4bca-453c-8072-77d6a0ceb89d-kube-api-access-s9xq8\") pod \"maas-api-revoke-sub-78594f64-9wwpt\" (UID: \"708051cf-4bca-453c-8072-77d6a0ceb89d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78594f64-9wwpt" Oct 30 21:34:11.625833 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.625813 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqlsv\" (UniqueName: \"kubernetes.io/projected/3f908467-b845-4a03-849d-4ab0c9ef698b-kube-api-access-gqlsv\") pod \"maas-api-revoke-sub-0b312f00-w9p96\" (UID: \"3f908467-b845-4a03-849d-4ab0c9ef698b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b312f00-w9p96" Oct 30 21:34:11.742363 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.742333 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b312f00-w9p96" Oct 30 21:34:11.749341 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:11.749318 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78594f64-9wwpt" Oct 30 21:34:12.023927 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:12.023904 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b312f00-w9p96"] Oct 30 21:34:12.026546 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:34:12.026518 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f908467_b845_4a03_849d_4ab0c9ef698b.slice/crio-cd6a5973b726a6f9d74ff8e115248da3419eea1d90860533a886ca64d61d6ef9 WatchSource:0}: Error finding container cd6a5973b726a6f9d74ff8e115248da3419eea1d90860533a886ca64d61d6ef9: Status 404 returned error can't find the container with id cd6a5973b726a6f9d74ff8e115248da3419eea1d90860533a886ca64d61d6ef9 Oct 30 21:34:12.028169 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:12.028140 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78594f64-9wwpt"] Oct 30 21:34:12.029736 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:34:12.029714 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708051cf_4bca_453c_8072_77d6a0ceb89d.slice/crio-75a7381d650260f1c238dc49872fea6572cd59fdad630e9f5eccc79968e61bbd WatchSource:0}: Error finding container 75a7381d650260f1c238dc49872fea6572cd59fdad630e9f5eccc79968e61bbd: Status 404 returned error can't find the container with id 75a7381d650260f1c238dc49872fea6572cd59fdad630e9f5eccc79968e61bbd Oct 30 21:34:12.127841 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:12.127806 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b312f00-w9p96" event={"ID":"3f908467-b845-4a03-849d-4ab0c9ef698b","Type":"ContainerStarted","Data":"cd6a5973b726a6f9d74ff8e115248da3419eea1d90860533a886ca64d61d6ef9"} Oct 30 21:34:12.129048 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:12.129024 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78594f64-9wwpt" event={"ID":"708051cf-4bca-453c-8072-77d6a0ceb89d","Type":"ContainerStarted","Data":"75a7381d650260f1c238dc49872fea6572cd59fdad630e9f5eccc79968e61bbd"} Oct 30 21:34:13.136382 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:13.136296 4848 generic.go:356] "Generic (PLEG): container finished" podID="3f908467-b845-4a03-849d-4ab0c9ef698b" containerID="8922894cfb8a432f514e5765ddee0fe3f16077136e20727baab850486e5ddbf3" exitCode=0 Oct 30 21:34:13.136821 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:13.136381 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b312f00-w9p96" event={"ID":"3f908467-b845-4a03-849d-4ab0c9ef698b","Type":"ContainerDied","Data":"8922894cfb8a432f514e5765ddee0fe3f16077136e20727baab850486e5ddbf3"} Oct 30 21:34:13.138469 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:13.138439 4848 generic.go:356] "Generic (PLEG): container finished" podID="708051cf-4bca-453c-8072-77d6a0ceb89d" containerID="752ec3765b9f768dc56c95f1fd0e500da8da79e604acddbbda64b13bb7c69588" exitCode=0 Oct 30 21:34:13.138589 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:13.138480 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78594f64-9wwpt" event={"ID":"708051cf-4bca-453c-8072-77d6a0ceb89d","Type":"ContainerDied","Data":"752ec3765b9f768dc56c95f1fd0e500da8da79e604acddbbda64b13bb7c69588"} Oct 30 21:34:14.336702 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:14.336668 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b312f00-w9p96" Oct 30 21:34:14.340044 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:14.340018 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78594f64-9wwpt" Oct 30 21:34:14.450947 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:14.450911 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708051cf-4bca-453c-8072-77d6a0ceb89d-kube-api-access-s9xq8\" (UniqueName: \"kubernetes.io/projected/708051cf-4bca-453c-8072-77d6a0ceb89d-kube-api-access-s9xq8\") pod \"708051cf-4bca-453c-8072-77d6a0ceb89d\" (UID: \"708051cf-4bca-453c-8072-77d6a0ceb89d\") " Oct 30 21:34:14.450947 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:14.450959 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f908467-b845-4a03-849d-4ab0c9ef698b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f908467-b845-4a03-849d-4ab0c9ef698b-maas-api-openshift-ca\") pod \"3f908467-b845-4a03-849d-4ab0c9ef698b\" (UID: \"3f908467-b845-4a03-849d-4ab0c9ef698b\") " Oct 30 21:34:14.451224 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:14.451049 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f908467-b845-4a03-849d-4ab0c9ef698b-kube-api-access-gqlsv\" (UniqueName: \"kubernetes.io/projected/3f908467-b845-4a03-849d-4ab0c9ef698b-kube-api-access-gqlsv\") pod \"3f908467-b845-4a03-849d-4ab0c9ef698b\" (UID: \"3f908467-b845-4a03-849d-4ab0c9ef698b\") " Oct 30 21:34:14.451224 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:14.451123 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708051cf-4bca-453c-8072-77d6a0ceb89d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708051cf-4bca-453c-8072-77d6a0ceb89d-maas-api-openshift-ca\") pod \"708051cf-4bca-453c-8072-77d6a0ceb89d\" (UID: \"708051cf-4bca-453c-8072-77d6a0ceb89d\") " Oct 30 21:34:14.451224 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:14.451157 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708051cf-4bca-453c-8072-77d6a0ceb89d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708051cf-4bca-453c-8072-77d6a0ceb89d-maas-api-xks-ca\") pod \"708051cf-4bca-453c-8072-77d6a0ceb89d\" (UID: \"708051cf-4bca-453c-8072-77d6a0ceb89d\") " Oct 30 21:34:14.451407 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:14.451226 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f908467-b845-4a03-849d-4ab0c9ef698b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f908467-b845-4a03-849d-4ab0c9ef698b-maas-api-xks-ca\") pod \"3f908467-b845-4a03-849d-4ab0c9ef698b\" (UID: \"3f908467-b845-4a03-849d-4ab0c9ef698b\") " Oct 30 21:34:14.451519 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:14.451489 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708051cf-4bca-453c-8072-77d6a0ceb89d-maas-api-openshift-ca" pod "708051cf-4bca-453c-8072-77d6a0ceb89d" (UID: "708051cf-4bca-453c-8072-77d6a0ceb89d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:34:14.451719 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:14.451691 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f908467-b845-4a03-849d-4ab0c9ef698b-maas-api-openshift-ca" pod "3f908467-b845-4a03-849d-4ab0c9ef698b" (UID: "3f908467-b845-4a03-849d-4ab0c9ef698b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:34:14.453518 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:14.453487 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708051cf-4bca-453c-8072-77d6a0ceb89d-maas-api-xks-ca" pod "708051cf-4bca-453c-8072-77d6a0ceb89d" (UID: "708051cf-4bca-453c-8072-77d6a0ceb89d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:14.453721 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:14.453700 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708051cf-4bca-453c-8072-77d6a0ceb89d-kube-api-access-s9xq8" pod "708051cf-4bca-453c-8072-77d6a0ceb89d" (UID: "708051cf-4bca-453c-8072-77d6a0ceb89d"). InnerVolumeSpecName "kube-api-access-s9xq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:14.453822 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:14.453733 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f908467-b845-4a03-849d-4ab0c9ef698b-kube-api-access-gqlsv" pod "3f908467-b845-4a03-849d-4ab0c9ef698b" (UID: "3f908467-b845-4a03-849d-4ab0c9ef698b"). InnerVolumeSpecName "kube-api-access-gqlsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:14.453979 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:14.453961 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f908467-b845-4a03-849d-4ab0c9ef698b-maas-api-xks-ca" pod "3f908467-b845-4a03-849d-4ab0c9ef698b" (UID: "3f908467-b845-4a03-849d-4ab0c9ef698b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:14.552862 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:14.552818 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708051cf-4bca-453c-8072-77d6a0ceb89d-maas-api-openshift-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:14.552862 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:14.552853 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708051cf-4bca-453c-8072-77d6a0ceb89d-maas-api-xks-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:14.552862 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:14.552863 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f908467-b845-4a03-849d-4ab0c9ef698b-maas-api-xks-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:14.552862 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:14.552872 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9xq8\" (UniqueName: \"kubernetes.io/projected/708051cf-4bca-453c-8072-77d6a0ceb89d-kube-api-access-s9xq8\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:14.553238 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:14.552881 4848 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f908467-b845-4a03-849d-4ab0c9ef698b-maas-api-openshift-ca\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:14.553238 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:14.552890 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqlsv\" (UniqueName: \"kubernetes.io/projected/3f908467-b845-4a03-849d-4ab0c9ef698b-kube-api-access-gqlsv\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:15.149692 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:15.149661 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b312f00-w9p96" Oct 30 21:34:15.149692 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:15.149674 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b312f00-w9p96" event={"ID":"3f908467-b845-4a03-849d-4ab0c9ef698b","Type":"ContainerDied","Data":"cd6a5973b726a6f9d74ff8e115248da3419eea1d90860533a886ca64d61d6ef9"} Oct 30 21:34:15.149966 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:15.149723 4848 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd6a5973b726a6f9d74ff8e115248da3419eea1d90860533a886ca64d61d6ef9" Oct 30 21:34:15.151048 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:15.151014 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78594f64-9wwpt" event={"ID":"708051cf-4bca-453c-8072-77d6a0ceb89d","Type":"ContainerDied","Data":"75a7381d650260f1c238dc49872fea6572cd59fdad630e9f5eccc79968e61bbd"} Oct 30 21:34:15.151048 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:15.151036 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78594f64-9wwpt" Oct 30 21:34:15.151232 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:15.151055 4848 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75a7381d650260f1c238dc49872fea6572cd59fdad630e9f5eccc79968e61bbd" Oct 30 21:34:22.509248 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:34:22.509211 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:34:26.694271 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:26.694230 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l"] Oct 30 21:34:26.697155 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:26.697129 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-317a33-l6j6l"] Oct 30 21:34:26.931268 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:26.931230 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="817ceb47-dc06-4c56-964a-d636f27dfefe" path="/var/lib/kubelet/pods/817ceb47-dc06-4c56-964a-d636f27dfefe/volumes" Oct 30 21:34:33.532237 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:33.532203 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24"] Oct 30 21:34:33.532667 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:33.532431 4848 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" podUID="9829d8aa-db34-42b9-bf27-316fa4cc86f1" containerName="istio-proxy" containerID="cri-o://6e794d063e146fd1b52b97fb8015449ba5401e431d304b0d7929ed8fa181d2e8" gracePeriod=30 Oct 30 21:34:34.732976 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:34.732946 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f767f5f8-dnjx6_dff97c85-a942-4a38-a233-a6563aa78d4d/authorino/0.log" Oct 30 21:34:35.275097 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:35.275060 4848 patch_prober.go:29] interesting pod/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:34:35.275257 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:35.275128 4848 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" podUID="9829d8aa-db34-42b9-bf27-316fa4cc86f1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:34:36.757173 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:36.757141 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b47bd7b99-gx47b_01739bcb-1df4-4118-9b9f-2d52db7ed86b/manager/0.log" Oct 30 21:34:36.978447 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:36.978415 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-chpzx_30fe2f63-49a7-43f5-9530-9f1cf25d635c/manager/0.log" Oct 30 21:34:37.916068 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:37.916030 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f5d6dd878-twxgq_e1ef8eca-79af-4e64-8cba-4b9caaa7e574/manager/0.log" Oct 30 21:34:38.098200 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.098167 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-htl5n_8c746478-bdc8-45c9-b5da-39a1fe98f1d1/server/0.log" Oct 30 21:34:38.272813 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.272784 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-btqgb_ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26/manager/0.log" Oct 30 21:34:38.467196 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.467162 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rf52l_7dce4323-1225-4bf8-b93e-a2a482b657a5/proxy/0.log" Oct 30 21:34:38.810503 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.810437 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 21:34:38.972742 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.972708 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-workload-certs\") pod \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " Oct 30 21:34:38.973183 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.972802 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-credential-socket\") pod \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " Oct 30 21:34:38.973183 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.972845 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istiod-ca-cert\") pod \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " Oct 30 21:34:38.973183 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.972894 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-podinfo\") pod \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " Oct 30 21:34:38.973183 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.972948 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-token\" (UniqueName: \"kubernetes.io/projected/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-token\") pod \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " Oct 30 21:34:38.973183 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.972974 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-workload-socket\") pod \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " Oct 30 21:34:38.973183 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.973000 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-envoy\") pod \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " Oct 30 21:34:38.973183 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.973024 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-workload-certs" pod "9829d8aa-db34-42b9-bf27-316fa4cc86f1" (UID: "9829d8aa-db34-42b9-bf27-316fa4cc86f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:34:38.973183 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.973049 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9829d8aa-db34-42b9-bf27-316fa4cc86f1-kube-api-access-vzc27\" (UniqueName: \"kubernetes.io/projected/9829d8aa-db34-42b9-bf27-316fa4cc86f1-kube-api-access-vzc27\") pod \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " Oct 30 21:34:38.973183 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.973082 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-credential-socket" pod "9829d8aa-db34-42b9-bf27-316fa4cc86f1" (UID: "9829d8aa-db34-42b9-bf27-316fa4cc86f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:34:38.973183 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.973152 4848 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-data\") pod \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\" (UID: \"9829d8aa-db34-42b9-bf27-316fa4cc86f1\") " Oct 30 21:34:38.973824 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.973372 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-workload-socket" pod "9829d8aa-db34-42b9-bf27-316fa4cc86f1" (UID: "9829d8aa-db34-42b9-bf27-316fa4cc86f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:34:38.973824 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.973403 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istiod-ca-cert" pod "9829d8aa-db34-42b9-bf27-316fa4cc86f1" (UID: "9829d8aa-db34-42b9-bf27-316fa4cc86f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:34:38.973824 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.973529 4848 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istiod-ca-cert\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:38.973824 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.973550 4848 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-workload-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:38.973824 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.973563 4848 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-workload-certs\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:38.973824 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.973576 4848 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-credential-socket\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:38.974123 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.973904 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-data" pod "9829d8aa-db34-42b9-bf27-316fa4cc86f1" (UID: "9829d8aa-db34-42b9-bf27-316fa4cc86f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:34:38.975432 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.975401 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-envoy" pod "9829d8aa-db34-42b9-bf27-316fa4cc86f1" (UID: "9829d8aa-db34-42b9-bf27-316fa4cc86f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:34:38.975716 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.975694 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-token" pod "9829d8aa-db34-42b9-bf27-316fa4cc86f1" (UID: "9829d8aa-db34-42b9-bf27-316fa4cc86f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:38.975849 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.975719 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9829d8aa-db34-42b9-bf27-316fa4cc86f1-kube-api-access-vzc27" pod "9829d8aa-db34-42b9-bf27-316fa4cc86f1" (UID: "9829d8aa-db34-42b9-bf27-316fa4cc86f1"). InnerVolumeSpecName "kube-api-access-vzc27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:38.975971 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:38.975885 4848 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-podinfo" pod "9829d8aa-db34-42b9-bf27-316fa4cc86f1" (UID: "9829d8aa-db34-42b9-bf27-316fa4cc86f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:34:39.074372 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:39.074282 4848 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-token\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:39.074372 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:39.074317 4848 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-envoy\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:39.074372 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:39.074329 4848 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzc27\" (UniqueName: \"kubernetes.io/projected/9829d8aa-db34-42b9-bf27-316fa4cc86f1-kube-api-access-vzc27\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:39.074372 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:39.074344 4848 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-data\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:39.074372 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:39.074355 4848 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9829d8aa-db34-42b9-bf27-316fa4cc86f1-istio-podinfo\") on node \"ip-10-0-130-52.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:34:39.193895 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:39.193864 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f767f5f8-dnjx6_dff97c85-a942-4a38-a233-a6563aa78d4d/authorino/0.log" Oct 30 21:34:39.263342 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:39.263312 4848 generic.go:356] "Generic (PLEG): container finished" podID="9829d8aa-db34-42b9-bf27-316fa4cc86f1" containerID="6e794d063e146fd1b52b97fb8015449ba5401e431d304b0d7929ed8fa181d2e8" exitCode=0 Oct 30 21:34:39.263550 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:39.263372 4848 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" Oct 30 21:34:39.263550 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:39.263399 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" event={"ID":"9829d8aa-db34-42b9-bf27-316fa4cc86f1","Type":"ContainerDied","Data":"6e794d063e146fd1b52b97fb8015449ba5401e431d304b0d7929ed8fa181d2e8"} Oct 30 21:34:39.263550 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:39.263456 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24" event={"ID":"9829d8aa-db34-42b9-bf27-316fa4cc86f1","Type":"ContainerDied","Data":"87bed5b15961bc7e983185ca6ac862ae65cffc9e00decfe56868dd53272ce3ca"} Oct 30 21:34:39.263550 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:39.263479 4848 scope.go:122] "RemoveContainer" containerID="6e794d063e146fd1b52b97fb8015449ba5401e431d304b0d7929ed8fa181d2e8" Oct 30 21:34:39.272893 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:39.272877 4848 scope.go:122] "RemoveContainer" containerID="6e794d063e146fd1b52b97fb8015449ba5401e431d304b0d7929ed8fa181d2e8" Oct 30 21:34:39.273167 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:34:39.273148 4848 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e794d063e146fd1b52b97fb8015449ba5401e431d304b0d7929ed8fa181d2e8\": container with ID starting with 6e794d063e146fd1b52b97fb8015449ba5401e431d304b0d7929ed8fa181d2e8 not found: ID does not exist" containerID="6e794d063e146fd1b52b97fb8015449ba5401e431d304b0d7929ed8fa181d2e8" Oct 30 21:34:39.273208 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:39.273177 4848 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e794d063e146fd1b52b97fb8015449ba5401e431d304b0d7929ed8fa181d2e8"} err="failed to get container status \"6e794d063e146fd1b52b97fb8015449ba5401e431d304b0d7929ed8fa181d2e8\": rpc error: code = NotFound desc = could not find container \"6e794d063e146fd1b52b97fb8015449ba5401e431d304b0d7929ed8fa181d2e8\": container with ID starting with 6e794d063e146fd1b52b97fb8015449ba5401e431d304b0d7929ed8fa181d2e8 not found: ID does not exist" Oct 30 21:34:39.285870 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:39.285828 4848 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24"] Oct 30 21:34:39.287685 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:39.287665 4848 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-317a33-openshift-default-6d6f98675-dvm24"] Oct 30 21:34:40.929902 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:40.929870 4848 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9829d8aa-db34-42b9-bf27-316fa4cc86f1" path="/var/lib/kubelet/pods/9829d8aa-db34-42b9-bf27-316fa4cc86f1/volumes" Oct 30 21:34:49.646604 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:49.646573 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b47bd7b99-gx47b_01739bcb-1df4-4118-9b9f-2d52db7ed86b/manager/0.log" Oct 30 21:34:49.832569 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:49.832538 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-chpzx_30fe2f63-49a7-43f5-9530-9f1cf25d635c/manager/0.log" Oct 30 21:34:50.716537 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:50.716504 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f5d6dd878-twxgq_e1ef8eca-79af-4e64-8cba-4b9caaa7e574/manager/0.log" Oct 30 21:34:50.891774 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:50.891744 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-htl5n_8c746478-bdc8-45c9-b5da-39a1fe98f1d1/server/0.log" Oct 30 21:34:51.075056 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:51.074981 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-btqgb_ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26/manager/0.log" Oct 30 21:34:51.287090 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:51.287056 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rf52l_7dce4323-1225-4bf8-b93e-a2a482b657a5/proxy/0.log" Oct 30 21:34:52.402984 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:52.402944 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0b312f00-w9p96_3f908467-b845-4a03-849d-4ab0c9ef698b/revoke-keys/0.log" Oct 30 21:34:52.509716 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:34:52.509670 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:34:52.954183 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:52.954158 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-78594f64-9wwpt_708051cf-4bca-453c-8072-77d6a0ceb89d/revoke-keys/0.log" Oct 30 21:34:54.209053 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:54.209027 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f21776fa-f824n_a88a7285-af5c-4693-93fb-8792d1f8959c/revoke-keys/0.log" Oct 30 21:34:54.390206 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:54.390180 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-b28mc_fe4fdd20-b075-45d5-ae3d-3959bfa3bb51/postgres/0.log" Oct 30 21:34:55.887237 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:55.887198 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f767f5f8-dnjx6_dff97c85-a942-4a38-a233-a6563aa78d4d/authorino/0.log" Oct 30 21:34:58.449716 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:58.449686 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64cbf78866-9kpj2_c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c/kube-auth-proxy/0.log" Oct 30 21:34:59.361192 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:59.361159 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-fmzmj_7dc1d3fc-a66d-41d9-b81c-d17801d8702e/istio-proxy/0.log" Oct 30 21:34:59.541873 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:59.541839 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-sb5c9_a6fc6562-45cb-47db-98ab-415e184c71b3/istio-proxy/0.log" Oct 30 21:34:59.918255 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:34:59.918190 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-shfhk_c7711661-bd4e-49ba-96f6-2739374b633e/payload-pre-processing/0.log" Oct 30 21:35:00.089241 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:00.089212 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fztgr_bce11e0b-618d-40b0-9709-d6b3deda2012/payload-processing/0.log" Oct 30 21:35:01.002117 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:01.002086 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-hzp87_70e69039-4f0f-4f23-81cf-a84eb0bbf942/main/0.log" Oct 30 21:35:01.356488 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:01.356382 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-vgkmj_d4d640ea-9cc3-4090-a65e-972250c29b28/main/0.log" Oct 30 21:35:01.891351 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:01.891321 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-gtxbn_15ef1876-cf8b-4383-9757-6569689a6978/main/0.log" Oct 30 21:35:02.064928 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:02.064895 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-d257f_ec7f6e4c-0b8f-4a83-bce8-5a0e4d937f76/main/0.log" Oct 30 21:35:12.918211 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.918173 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7gfd4/must-gather-c74bj"] Oct 30 21:35:12.918793 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.918751 4848 cpu_manager.go:414] "containerMap: removing container" podUID="9829d8aa-db34-42b9-bf27-316fa4cc86f1" containerName="istio-proxy" Oct 30 21:35:12.918858 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.918790 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9829d8aa-db34-42b9-bf27-316fa4cc86f1" containerName="istio-proxy" Oct 30 21:35:12.918858 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.918843 4848 cpu_manager.go:414] "containerMap: removing container" podUID="708051cf-4bca-453c-8072-77d6a0ceb89d" containerName="revoke-keys" Oct 30 21:35:12.918858 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.918851 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708051cf-4bca-453c-8072-77d6a0ceb89d" containerName="revoke-keys" Oct 30 21:35:12.918956 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.918865 4848 cpu_manager.go:414] "containerMap: removing container" podUID="3f908467-b845-4a03-849d-4ab0c9ef698b" containerName="revoke-keys" Oct 30 21:35:12.918956 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.918873 4848 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f908467-b845-4a03-849d-4ab0c9ef698b" containerName="revoke-keys" Oct 30 21:35:12.919027 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.918962 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f908467-b845-4a03-849d-4ab0c9ef698b" containerName="revoke-keys" Oct 30 21:35:12.919027 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.918980 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="9829d8aa-db34-42b9-bf27-316fa4cc86f1" containerName="istio-proxy" Oct 30 21:35:12.919027 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.918996 4848 memory_manager.go:365] "RemoveStaleState removing state" podUID="708051cf-4bca-453c-8072-77d6a0ceb89d" containerName="revoke-keys" Oct 30 21:35:12.922265 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.922242 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pp828/must-gather-w4spj"] Oct 30 21:35:12.922393 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.922375 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7gfd4/must-gather-c74bj" Oct 30 21:35:12.925755 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.925731 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pp828/must-gather-w4spj" Oct 30 21:35:12.930489 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.930464 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7gfd4\"/\"kube-root-ca.crt\"" Oct 30 21:35:12.930617 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.930548 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7gfd4\"/\"openshift-service-ca.crt\"" Oct 30 21:35:12.931434 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.931411 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7gfd4\"/\"default-dockercfg-sn95d\"" Oct 30 21:35:12.934308 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.934286 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7gfd4/must-gather-c74bj"] Oct 30 21:35:12.934689 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.934667 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pp828/must-gather-w4spj"] Oct 30 21:35:12.937846 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.937737 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pp828\"/\"kube-root-ca.crt\"" Oct 30 21:35:12.937846 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.937818 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pp828\"/\"openshift-service-ca.crt\"" Oct 30 21:35:12.959918 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.959879 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pp828\"/\"default-dockercfg-72zhz\"" Oct 30 21:35:12.981849 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.981817 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/25c5b15d-ee4b-44a6-a67b-ee5d21a6c40a-must-gather-output\") pod \"must-gather-c74bj\" (UID: \"25c5b15d-ee4b-44a6-a67b-ee5d21a6c40a\") " pod="openshift-must-gather-7gfd4/must-gather-c74bj" Oct 30 21:35:12.982005 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:12.981973 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn7f9\" (UniqueName: \"kubernetes.io/projected/25c5b15d-ee4b-44a6-a67b-ee5d21a6c40a-kube-api-access-kn7f9\") pod \"must-gather-c74bj\" (UID: \"25c5b15d-ee4b-44a6-a67b-ee5d21a6c40a\") " pod="openshift-must-gather-7gfd4/must-gather-c74bj" Oct 30 21:35:13.012020 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.011991 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8ln49/must-gather-5cjn5"] Oct 30 21:35:13.015479 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.015461 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8ln49/must-gather-5cjn5" Oct 30 21:35:13.023708 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.023678 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8ln49/must-gather-5cjn5"] Oct 30 21:35:13.029666 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.029644 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8ln49\"/\"kube-root-ca.crt\"" Oct 30 21:35:13.029666 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.029661 4848 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8ln49\"/\"openshift-service-ca.crt\"" Oct 30 21:35:13.029867 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.029842 4848 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8ln49\"/\"default-dockercfg-n87st\"" Oct 30 21:35:13.083296 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.083273 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn7f9\" (UniqueName: \"kubernetes.io/projected/25c5b15d-ee4b-44a6-a67b-ee5d21a6c40a-kube-api-access-kn7f9\") pod \"must-gather-c74bj\" (UID: \"25c5b15d-ee4b-44a6-a67b-ee5d21a6c40a\") " pod="openshift-must-gather-7gfd4/must-gather-c74bj" Oct 30 21:35:13.083430 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.083320 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/25c5b15d-ee4b-44a6-a67b-ee5d21a6c40a-must-gather-output\") pod \"must-gather-c74bj\" (UID: \"25c5b15d-ee4b-44a6-a67b-ee5d21a6c40a\") " pod="openshift-must-gather-7gfd4/must-gather-c74bj" Oct 30 21:35:13.083430 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.083343 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22czg\" (UniqueName: \"kubernetes.io/projected/431ae25d-22c6-49fc-b5f0-2060c6fb6c3a-kube-api-access-22czg\") pod \"must-gather-w4spj\" (UID: \"431ae25d-22c6-49fc-b5f0-2060c6fb6c3a\") " pod="openshift-must-gather-pp828/must-gather-w4spj" Oct 30 21:35:13.083430 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.083381 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/431ae25d-22c6-49fc-b5f0-2060c6fb6c3a-must-gather-output\") pod \"must-gather-w4spj\" (UID: \"431ae25d-22c6-49fc-b5f0-2060c6fb6c3a\") " pod="openshift-must-gather-pp828/must-gather-w4spj" Oct 30 21:35:13.083430 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.083409 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgpc2\" (UniqueName: \"kubernetes.io/projected/7ad123a6-e184-49de-a155-15e7c060ecbf-kube-api-access-hgpc2\") pod \"must-gather-5cjn5\" (UID: \"7ad123a6-e184-49de-a155-15e7c060ecbf\") " pod="openshift-must-gather-8ln49/must-gather-5cjn5" Oct 30 21:35:13.083597 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.083545 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ad123a6-e184-49de-a155-15e7c060ecbf-must-gather-output\") pod \"must-gather-5cjn5\" (UID: \"7ad123a6-e184-49de-a155-15e7c060ecbf\") " pod="openshift-must-gather-8ln49/must-gather-5cjn5" Oct 30 21:35:13.083649 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.083634 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/25c5b15d-ee4b-44a6-a67b-ee5d21a6c40a-must-gather-output\") pod \"must-gather-c74bj\" (UID: \"25c5b15d-ee4b-44a6-a67b-ee5d21a6c40a\") " pod="openshift-must-gather-7gfd4/must-gather-c74bj" Oct 30 21:35:13.094271 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.094243 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn7f9\" (UniqueName: \"kubernetes.io/projected/25c5b15d-ee4b-44a6-a67b-ee5d21a6c40a-kube-api-access-kn7f9\") pod \"must-gather-c74bj\" (UID: \"25c5b15d-ee4b-44a6-a67b-ee5d21a6c40a\") " pod="openshift-must-gather-7gfd4/must-gather-c74bj" Oct 30 21:35:13.184393 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.184357 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ad123a6-e184-49de-a155-15e7c060ecbf-must-gather-output\") pod \"must-gather-5cjn5\" (UID: \"7ad123a6-e184-49de-a155-15e7c060ecbf\") " pod="openshift-must-gather-8ln49/must-gather-5cjn5" Oct 30 21:35:13.184534 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.184452 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22czg\" (UniqueName: \"kubernetes.io/projected/431ae25d-22c6-49fc-b5f0-2060c6fb6c3a-kube-api-access-22czg\") pod \"must-gather-w4spj\" (UID: \"431ae25d-22c6-49fc-b5f0-2060c6fb6c3a\") " pod="openshift-must-gather-pp828/must-gather-w4spj" Oct 30 21:35:13.184534 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.184488 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/431ae25d-22c6-49fc-b5f0-2060c6fb6c3a-must-gather-output\") pod \"must-gather-w4spj\" (UID: \"431ae25d-22c6-49fc-b5f0-2060c6fb6c3a\") " pod="openshift-must-gather-pp828/must-gather-w4spj" Oct 30 21:35:13.184534 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.184516 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgpc2\" (UniqueName: \"kubernetes.io/projected/7ad123a6-e184-49de-a155-15e7c060ecbf-kube-api-access-hgpc2\") pod \"must-gather-5cjn5\" (UID: \"7ad123a6-e184-49de-a155-15e7c060ecbf\") " pod="openshift-must-gather-8ln49/must-gather-5cjn5" Oct 30 21:35:13.184725 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.184668 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7ad123a6-e184-49de-a155-15e7c060ecbf-must-gather-output\") pod \"must-gather-5cjn5\" (UID: \"7ad123a6-e184-49de-a155-15e7c060ecbf\") " pod="openshift-must-gather-8ln49/must-gather-5cjn5" Oct 30 21:35:13.184810 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.184762 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/431ae25d-22c6-49fc-b5f0-2060c6fb6c3a-must-gather-output\") pod \"must-gather-w4spj\" (UID: \"431ae25d-22c6-49fc-b5f0-2060c6fb6c3a\") " pod="openshift-must-gather-pp828/must-gather-w4spj" Oct 30 21:35:13.196001 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.195979 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22czg\" (UniqueName: \"kubernetes.io/projected/431ae25d-22c6-49fc-b5f0-2060c6fb6c3a-kube-api-access-22czg\") pod \"must-gather-w4spj\" (UID: \"431ae25d-22c6-49fc-b5f0-2060c6fb6c3a\") " pod="openshift-must-gather-pp828/must-gather-w4spj" Oct 30 21:35:13.196105 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.196086 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgpc2\" (UniqueName: \"kubernetes.io/projected/7ad123a6-e184-49de-a155-15e7c060ecbf-kube-api-access-hgpc2\") pod \"must-gather-5cjn5\" (UID: \"7ad123a6-e184-49de-a155-15e7c060ecbf\") " pod="openshift-must-gather-8ln49/must-gather-5cjn5" Oct 30 21:35:13.242688 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.242668 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7gfd4/must-gather-c74bj" Oct 30 21:35:13.249737 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.249719 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pp828/must-gather-w4spj" Oct 30 21:35:13.330343 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.330301 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8ln49/must-gather-5cjn5" Oct 30 21:35:13.521303 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.521280 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7gfd4/must-gather-c74bj"] Oct 30 21:35:13.523948 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:35:13.523920 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25c5b15d_ee4b_44a6_a67b_ee5d21a6c40a.slice/crio-90af192783088aa44326b2ee711d24bc17317f851136c4f2af74939b0550f16a WatchSource:0}: Error finding container 90af192783088aa44326b2ee711d24bc17317f851136c4f2af74939b0550f16a: Status 404 returned error can't find the container with id 90af192783088aa44326b2ee711d24bc17317f851136c4f2af74939b0550f16a Oct 30 21:35:13.525175 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.525151 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pp828/must-gather-w4spj"] Oct 30 21:35:13.525962 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.525941 4848 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:35:13.526384 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:35:13.526365 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod431ae25d_22c6_49fc_b5f0_2060c6fb6c3a.slice/crio-59c9de1b23b36f1089d1af9164f78f2fa5c21daafec8ae6096f17e4ef37322e2 WatchSource:0}: Error finding container 59c9de1b23b36f1089d1af9164f78f2fa5c21daafec8ae6096f17e4ef37322e2: Status 404 returned error can't find the container with id 59c9de1b23b36f1089d1af9164f78f2fa5c21daafec8ae6096f17e4ef37322e2 Oct 30 21:35:13.608344 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:13.608320 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8ln49/must-gather-5cjn5"] Oct 30 21:35:13.610131 ip-10-0-130-52 kubenswrapper[4848]: W1030 21:35:13.610106 4848 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ad123a6_e184_49de_a155_15e7c060ecbf.slice/crio-6fd156431108f0d669a532b7fdd062f720fa2102d3337a97a607cbfa891cbd5f WatchSource:0}: Error finding container 6fd156431108f0d669a532b7fdd062f720fa2102d3337a97a607cbfa891cbd5f: Status 404 returned error can't find the container with id 6fd156431108f0d669a532b7fdd062f720fa2102d3337a97a607cbfa891cbd5f Oct 30 21:35:14.424408 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:14.424380 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8ln49/must-gather-5cjn5" event={"ID":"7ad123a6-e184-49de-a155-15e7c060ecbf","Type":"ContainerStarted","Data":"6fd156431108f0d669a532b7fdd062f720fa2102d3337a97a607cbfa891cbd5f"} Oct 30 21:35:14.425683 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:14.425657 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7gfd4/must-gather-c74bj" event={"ID":"25c5b15d-ee4b-44a6-a67b-ee5d21a6c40a","Type":"ContainerStarted","Data":"90af192783088aa44326b2ee711d24bc17317f851136c4f2af74939b0550f16a"} Oct 30 21:35:14.426781 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:14.426742 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pp828/must-gather-w4spj" event={"ID":"431ae25d-22c6-49fc-b5f0-2060c6fb6c3a","Type":"ContainerStarted","Data":"59c9de1b23b36f1089d1af9164f78f2fa5c21daafec8ae6096f17e4ef37322e2"} Oct 30 21:35:15.469478 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:15.469421 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8ln49/must-gather-5cjn5" event={"ID":"7ad123a6-e184-49de-a155-15e7c060ecbf","Type":"ContainerStarted","Data":"77ca1a9c90b7c2f89dd1a83ecf4da136831c17d065f81d8aa5c1df3e0653765f"} Oct 30 21:35:15.470152 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:15.469488 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8ln49/must-gather-5cjn5" event={"ID":"7ad123a6-e184-49de-a155-15e7c060ecbf","Type":"ContainerStarted","Data":"5f3e2d734ece7c3559175f30f04f135402054448dc58e743f45b40bd3b48ccc3"} Oct 30 21:35:15.486615 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:15.486556 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8ln49/must-gather-5cjn5" podStartSLOduration=2.7607324 podStartE2EDuration="3.486537669s" totalImagesPullingTime="725.805269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:15.485048155 +0000 UTC m=+3420.664944676" watchObservedRunningTime="2026-10-30 21:35:15.486537669 +0000 UTC m=+3420.666434177" Oct 30 21:35:17.556805 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:17.556660 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5wp5n_a99706d1-d012-4dd9-b46a-5ec139287351/global-pull-secret-syncer/0.log" Oct 30 21:35:17.929264 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:17.929228 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-grp6n_4c57ecae-89cb-4901-8f9e-e4586771f5d8/konnectivity-agent/0.log" Oct 30 21:35:18.057292 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:18.057259 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-52.us-east-2.compute.internal_879a456e73ba77e5a50d63568935393e/haproxy/0.log" Oct 30 21:35:22.575790 ip-10-0-130-52 kubenswrapper[4848]: E1030 21:35:22.570532 4848 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k4d59" containerName="extractor" Oct 30 21:35:26.843268 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:26.842200 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f767f5f8-dnjx6_dff97c85-a942-4a38-a233-a6563aa78d4d/authorino/0.log" Oct 30 21:35:29.103056 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:29.102968 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b8fa08e-4def-48af-9546-903b17761894/alertmanager/0.log" Oct 30 21:35:29.134113 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:29.134061 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b8fa08e-4def-48af-9546-903b17761894/config-reloader/0.log" Oct 30 21:35:29.174376 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:29.174345 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b8fa08e-4def-48af-9546-903b17761894/kube-rbac-proxy-web/0.log" Oct 30 21:35:29.231939 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:29.231880 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b8fa08e-4def-48af-9546-903b17761894/kube-rbac-proxy/0.log" Oct 30 21:35:29.259837 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:29.259808 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b8fa08e-4def-48af-9546-903b17761894/kube-rbac-proxy-metric/0.log" Oct 30 21:35:29.288653 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:29.288629 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b8fa08e-4def-48af-9546-903b17761894/prom-label-proxy/0.log" Oct 30 21:35:29.323816 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:29.323782 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6b8fa08e-4def-48af-9546-903b17761894/init-config-reloader/0.log" Oct 30 21:35:29.572836 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:29.572699 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bqhgw_a97f29d4-5d15-432a-a493-6011cf8e059e/node-exporter/0.log" Oct 30 21:35:29.600460 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:29.600428 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bqhgw_a97f29d4-5d15-432a-a493-6011cf8e059e/kube-rbac-proxy/0.log" Oct 30 21:35:29.630531 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:29.630501 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bqhgw_a97f29d4-5d15-432a-a493-6011cf8e059e/init-textfile/0.log" Oct 30 21:35:29.875657 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:29.875414 4848 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n"] Oct 30 21:35:29.942609 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:29.941869 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n"] Oct 30 21:35:29.942609 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:29.941959 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" Oct 30 21:35:30.022529 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:30.022459 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ce58c7f-2401-465b-a6e8-3616fcd24d30-sys\") pod \"perf-node-gather-daemonset-2jv7n\" (UID: \"3ce58c7f-2401-465b-a6e8-3616fcd24d30\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" Oct 30 21:35:30.022529 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:30.022504 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3ce58c7f-2401-465b-a6e8-3616fcd24d30-podres\") pod \"perf-node-gather-daemonset-2jv7n\" (UID: \"3ce58c7f-2401-465b-a6e8-3616fcd24d30\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" Oct 30 21:35:30.022750 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:30.022631 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3ce58c7f-2401-465b-a6e8-3616fcd24d30-proc\") pod \"perf-node-gather-daemonset-2jv7n\" (UID: \"3ce58c7f-2401-465b-a6e8-3616fcd24d30\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" Oct 30 21:35:30.022750 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:30.022698 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt6ph\" (UniqueName: \"kubernetes.io/projected/3ce58c7f-2401-465b-a6e8-3616fcd24d30-kube-api-access-qt6ph\") pod \"perf-node-gather-daemonset-2jv7n\" (UID: \"3ce58c7f-2401-465b-a6e8-3616fcd24d30\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" Oct 30 21:35:30.023187 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:30.023080 4848 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ce58c7f-2401-465b-a6e8-3616fcd24d30-lib-modules\") pod \"perf-node-gather-daemonset-2jv7n\" (UID: \"3ce58c7f-2401-465b-a6e8-3616fcd24d30\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" Oct 30 21:35:30.126829 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:30.124487 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ce58c7f-2401-465b-a6e8-3616fcd24d30-lib-modules\") pod \"perf-node-gather-daemonset-2jv7n\" (UID: \"3ce58c7f-2401-465b-a6e8-3616fcd24d30\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" Oct 30 21:35:30.126829 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:30.124536 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ce58c7f-2401-465b-a6e8-3616fcd24d30-sys\") pod \"perf-node-gather-daemonset-2jv7n\" (UID: \"3ce58c7f-2401-465b-a6e8-3616fcd24d30\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" Oct 30 21:35:30.126829 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:30.124567 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3ce58c7f-2401-465b-a6e8-3616fcd24d30-podres\") pod \"perf-node-gather-daemonset-2jv7n\" (UID: \"3ce58c7f-2401-465b-a6e8-3616fcd24d30\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" Oct 30 21:35:30.126829 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:30.124600 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3ce58c7f-2401-465b-a6e8-3616fcd24d30-proc\") pod \"perf-node-gather-daemonset-2jv7n\" (UID: \"3ce58c7f-2401-465b-a6e8-3616fcd24d30\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" Oct 30 21:35:30.126829 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:30.124639 4848 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt6ph\" (UniqueName: \"kubernetes.io/projected/3ce58c7f-2401-465b-a6e8-3616fcd24d30-kube-api-access-qt6ph\") pod \"perf-node-gather-daemonset-2jv7n\" (UID: \"3ce58c7f-2401-465b-a6e8-3616fcd24d30\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" Oct 30 21:35:30.126829 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:30.125277 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3ce58c7f-2401-465b-a6e8-3616fcd24d30-podres\") pod \"perf-node-gather-daemonset-2jv7n\" (UID: \"3ce58c7f-2401-465b-a6e8-3616fcd24d30\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" Oct 30 21:35:30.126829 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:30.125425 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ce58c7f-2401-465b-a6e8-3616fcd24d30-lib-modules\") pod \"perf-node-gather-daemonset-2jv7n\" (UID: \"3ce58c7f-2401-465b-a6e8-3616fcd24d30\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" Oct 30 21:35:30.126829 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:30.125481 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3ce58c7f-2401-465b-a6e8-3616fcd24d30-proc\") pod \"perf-node-gather-daemonset-2jv7n\" (UID: \"3ce58c7f-2401-465b-a6e8-3616fcd24d30\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" Oct 30 21:35:30.126829 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:30.125487 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ce58c7f-2401-465b-a6e8-3616fcd24d30-sys\") pod \"perf-node-gather-daemonset-2jv7n\" (UID: \"3ce58c7f-2401-465b-a6e8-3616fcd24d30\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" Oct 30 21:35:30.136503 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:30.136472 4848 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt6ph\" (UniqueName: \"kubernetes.io/projected/3ce58c7f-2401-465b-a6e8-3616fcd24d30-kube-api-access-qt6ph\") pod \"perf-node-gather-daemonset-2jv7n\" (UID: \"3ce58c7f-2401-465b-a6e8-3616fcd24d30\") " pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" Oct 30 21:35:30.317700 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:30.317666 4848 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" Oct 30 21:35:30.671588 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:30.663260 4848 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n"] Oct 30 21:35:30.824171 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:30.820924 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" event={"ID":"3ce58c7f-2401-465b-a6e8-3616fcd24d30","Type":"ContainerStarted","Data":"9fd0473e7583fc7e52f49455cf13ee07b0cb8c2069385b43f1b7cc99230e1a51"} Oct 30 21:35:31.832389 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:31.832342 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" event={"ID":"3ce58c7f-2401-465b-a6e8-3616fcd24d30","Type":"ContainerStarted","Data":"e15b432a8ecf56525f285c1055b884b763935691a434b1b496919a2bcb2f095f"} Oct 30 21:35:31.832825 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:31.832633 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" Oct 30 21:35:32.865098 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:32.865056 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pp828/must-gather-w4spj" event={"ID":"431ae25d-22c6-49fc-b5f0-2060c6fb6c3a","Type":"ContainerStarted","Data":"5614d84c243622d3c1d380806faadf1d35ec51f14262b9a6197bea1cea98bde1"} Oct 30 21:35:32.865668 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:32.865646 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pp828/must-gather-w4spj" event={"ID":"431ae25d-22c6-49fc-b5f0-2060c6fb6c3a","Type":"ContainerStarted","Data":"d07f9c5e490d92fb779c2b6100711e7fe726b9b9be1c95cf63461ffb1470c30d"} Oct 30 21:35:32.884508 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:32.884226 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7gfd4/must-gather-c74bj" event={"ID":"25c5b15d-ee4b-44a6-a67b-ee5d21a6c40a","Type":"ContainerStarted","Data":"3051624afa7354e51fd211e629baade248199162724407b5cfed818c78bb1a6a"} Oct 30 21:35:32.884508 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:32.884270 4848 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7gfd4/must-gather-c74bj" event={"ID":"25c5b15d-ee4b-44a6-a67b-ee5d21a6c40a","Type":"ContainerStarted","Data":"917e021781bfd14aeae3ee604896bb2061e3e8b576deff51db23192d4465e9de"} Oct 30 21:35:32.889606 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:32.888365 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pp828/must-gather-w4spj" podStartSLOduration=2.422417729 podStartE2EDuration="20.888348886s" totalImagesPullingTime="18.465931157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:32.884668808 +0000 UTC m=+3438.064565315" watchObservedRunningTime="2026-10-30 21:35:32.888348886 +0000 UTC m=+3438.068245394" Oct 30 21:35:32.889606 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:32.888488 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" podStartSLOduration=3.888476899 podStartE2EDuration="3.888476899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:31.854233617 +0000 UTC m=+3437.034130125" watchObservedRunningTime="2026-10-30 21:35:32.888476899 +0000 UTC m=+3438.068373407" Oct 30 21:35:33.589520 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:33.589489 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-59d56c645b-72gc5_f311652d-e8bf-4256-87cb-ec157cdf73c3/console/0.log" Oct 30 21:35:33.984851 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:33.984793 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b47bd7b99-gx47b_01739bcb-1df4-4118-9b9f-2d52db7ed86b/manager/0.log" Oct 30 21:35:33.999977 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:33.999926 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-chpzx_30fe2f63-49a7-43f5-9530-9f1cf25d635c/manager/0.log" Oct 30 21:35:34.022266 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:34.022109 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b47bd7b99-gx47b_01739bcb-1df4-4118-9b9f-2d52db7ed86b/manager/0.log" Oct 30 21:35:34.036661 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:34.036620 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-chpzx_30fe2f63-49a7-43f5-9530-9f1cf25d635c/manager/0.log" Oct 30 21:35:35.808219 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:35.808183 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f5d6dd878-twxgq_e1ef8eca-79af-4e64-8cba-4b9caaa7e574/manager/0.log" Oct 30 21:35:35.841787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:35.839567 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f5d6dd878-twxgq_e1ef8eca-79af-4e64-8cba-4b9caaa7e574/manager/0.log" Oct 30 21:35:35.841787 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:35.840098 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-htl5n_8c746478-bdc8-45c9-b5da-39a1fe98f1d1/server/0.log" Oct 30 21:35:35.858061 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:35.856950 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-htl5n_8c746478-bdc8-45c9-b5da-39a1fe98f1d1/server/0.log" Oct 30 21:35:36.755539 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:36.755508 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tvdg8_b66dabb7-8606-4898-99ca-954ffd602da2/dns/0.log" Oct 30 21:35:36.794059 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:36.794011 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tvdg8_b66dabb7-8606-4898-99ca-954ffd602da2/kube-rbac-proxy/0.log" Oct 30 21:35:36.903658 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:36.903629 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vcndb_ab625ab1-de4f-4418-bd65-edb628d963fe/dns-node-resolver/0.log" Oct 30 21:35:37.486271 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:37.486142 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-btqgb_ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26/manager/0.log" Oct 30 21:35:37.512986 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:37.512952 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-btqgb_ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26/manager/0.log" Oct 30 21:35:37.575623 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:37.575587 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rf52l_7dce4323-1225-4bf8-b93e-a2a482b657a5/proxy/0.log" Oct 30 21:35:37.582666 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:37.582625 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rf52l_7dce4323-1225-4bf8-b93e-a2a482b657a5/proxy/0.log" Oct 30 21:35:37.897169 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:37.897095 4848 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8ln49/perf-node-gather-daemonset-2jv7n" Oct 30 21:35:37.926412 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:37.926353 4848 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7gfd4/must-gather-c74bj" podStartSLOduration=7.4741730650000004 podStartE2EDuration="25.926335744s" totalImagesPullingTime="18.452162679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:32.902975854 +0000 UTC m=+3438.082872364" watchObservedRunningTime="2026-10-30 21:35:37.926335744 +0000 UTC m=+3443.106232252" Oct 30 21:35:37.944496 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:37.944471 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-554jb_da9ceed2-3ab8-43dc-88a3-f20d7351ac30/node-ca/0.log" Oct 30 21:35:39.037691 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:39.037660 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b47bd7b99-gx47b_01739bcb-1df4-4118-9b9f-2d52db7ed86b/manager/0.log" Oct 30 21:35:39.056126 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:39.056100 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-chpzx_30fe2f63-49a7-43f5-9530-9f1cf25d635c/manager/0.log" Oct 30 21:35:39.145052 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:39.145025 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b47bd7b99-gx47b_01739bcb-1df4-4118-9b9f-2d52db7ed86b/manager/0.log" Oct 30 21:35:39.168371 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:39.168347 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-chpzx_30fe2f63-49a7-43f5-9530-9f1cf25d635c/manager/0.log" Oct 30 21:35:40.827153 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:40.827121 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f5d6dd878-twxgq_e1ef8eca-79af-4e64-8cba-4b9caaa7e574/manager/0.log" Oct 30 21:35:40.891903 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:40.891871 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-htl5n_8c746478-bdc8-45c9-b5da-39a1fe98f1d1/server/0.log" Oct 30 21:35:40.908065 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:40.907978 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f5d6dd878-twxgq_e1ef8eca-79af-4e64-8cba-4b9caaa7e574/manager/0.log" Oct 30 21:35:40.962628 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:40.962591 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-htl5n_8c746478-bdc8-45c9-b5da-39a1fe98f1d1/server/0.log" Oct 30 21:35:42.612673 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:42.612639 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-btqgb_ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26/manager/0.log" Oct 30 21:35:42.668235 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:42.668202 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-btqgb_ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26/manager/0.log" Oct 30 21:35:42.672692 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:42.672666 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rf52l_7dce4323-1225-4bf8-b93e-a2a482b657a5/proxy/0.log" Oct 30 21:35:42.738524 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:42.738446 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rf52l_7dce4323-1225-4bf8-b93e-a2a482b657a5/proxy/0.log" Oct 30 21:35:43.288799 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:43.288362 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64cbf78866-9kpj2_c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c/kube-auth-proxy/0.log" Oct 30 21:35:43.490294 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:43.490265 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-fmzmj_7dc1d3fc-a66d-41d9-b81c-d17801d8702e/istio-proxy/0.log" Oct 30 21:35:43.530927 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:43.529234 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-sb5c9_a6fc6562-45cb-47db-98ab-415e184c71b3/istio-proxy/0.log" Oct 30 21:35:43.595397 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:43.595323 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-shfhk_c7711661-bd4e-49ba-96f6-2739374b633e/payload-pre-processing/0.log" Oct 30 21:35:43.641052 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:43.641018 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fztgr_bce11e0b-618d-40b0-9709-d6b3deda2012/payload-processing/0.log" Oct 30 21:35:44.187270 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:44.187240 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b47bd7b99-gx47b_01739bcb-1df4-4118-9b9f-2d52db7ed86b/manager/0.log" Oct 30 21:35:44.201235 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:44.201192 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-chpzx_30fe2f63-49a7-43f5-9530-9f1cf25d635c/manager/0.log" Oct 30 21:35:44.298089 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:44.298056 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-b47bd7b99-gx47b_01739bcb-1df4-4118-9b9f-2d52db7ed86b/manager/0.log" Oct 30 21:35:44.317229 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:44.316819 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-chpzx_30fe2f63-49a7-43f5-9530-9f1cf25d635c/manager/0.log" Oct 30 21:35:46.214087 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:46.208520 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f5d6dd878-twxgq_e1ef8eca-79af-4e64-8cba-4b9caaa7e574/manager/0.log" Oct 30 21:35:46.243063 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:46.243025 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-htl5n_8c746478-bdc8-45c9-b5da-39a1fe98f1d1/server/0.log" Oct 30 21:35:46.366730 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:46.366653 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f5d6dd878-twxgq_e1ef8eca-79af-4e64-8cba-4b9caaa7e574/manager/0.log" Oct 30 21:35:46.432287 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:46.432239 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-htl5n_8c746478-bdc8-45c9-b5da-39a1fe98f1d1/server/0.log" Oct 30 21:35:46.636654 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:46.636559 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64cbf78866-9kpj2_c3ac8ee6-0f8b-40db-b97c-cd9ecf694f6c/kube-auth-proxy/0.log" Oct 30 21:35:46.782553 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:46.782518 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-fmzmj_7dc1d3fc-a66d-41d9-b81c-d17801d8702e/istio-proxy/0.log" Oct 30 21:35:46.817530 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:46.817506 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-sb5c9_a6fc6562-45cb-47db-98ab-415e184c71b3/istio-proxy/0.log" Oct 30 21:35:46.863307 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:46.863270 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-shfhk_c7711661-bd4e-49ba-96f6-2739374b633e/payload-pre-processing/0.log" Oct 30 21:35:46.873812 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:46.873657 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9w8gw_a8e2605f-358c-4fd1-ae8a-524282c76f72/serve-healthcheck-canary/0.log" Oct 30 21:35:46.891026 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:46.890959 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fztgr_bce11e0b-618d-40b0-9709-d6b3deda2012/payload-processing/0.log" Oct 30 21:35:47.910956 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:47.910853 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k4d59_dbe199fd-1f79-4fa3-9136-21a58fa1202d/kube-rbac-proxy/0.log" Oct 30 21:35:47.944746 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:47.944708 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k4d59_dbe199fd-1f79-4fa3-9136-21a58fa1202d/exporter/0.log" Oct 30 21:35:47.950471 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:47.950448 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-btqgb_ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26/manager/0.log" Oct 30 21:35:47.976172 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:47.976146 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k4d59_dbe199fd-1f79-4fa3-9136-21a58fa1202d/extractor/0.log" Oct 30 21:35:48.022005 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:48.021974 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rf52l_7dce4323-1225-4bf8-b93e-a2a482b657a5/proxy/0.log" Oct 30 21:35:48.105717 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:48.105678 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-btqgb_ab0ac0d4-5487-4f29-8a5a-ffe74ef87d26/manager/0.log" Oct 30 21:35:48.183272 ip-10-0-130-52 kubenswrapper[4848]: I1030 21:35:48.183244 4848 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-rf52l_7dce4323-1225-4bf8-b93e-a2a482b657a5/proxy/0.log"